{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IERG 6130 Assignment 3: Policy Gradient\n",
    "\n",
    "*2019-2020 2nd term, IERG 6130: Reinforcement Learning and Beyond. Department of Information Engineering, The Chinese University of Hong Kong. Course Instructor: Professor ZHOU Bolei. Assignment author: PENG Zhenghao.*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Student Name | Student ID |\n",
    "| :----: | :----: |\n",
    "| ZhenboWang | None |\n",
    "\n",
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welecome to the assignment 3 of our RL course. \n",
    "\n",
    "You need to go through this self-contained notebook, which contains many TODOs in part of the cells and has special `[TODO]` signs. You need to finish all TODOs. Some of them may be easy such as uncommenting a line, some of them may be difficult such as implementing a function. You can find them by searching the `[TODO]` symbol. However, we suggest you to go through the notebook step by step, which would give you a better understanding of the content.\n",
    "\n",
    "You are encouraged to add more code on extra cells at the end of the each section to investigate the problems you think interesting. At the end of the file, we left a place for you to optionaly write comments.\n",
    "\n",
    "Please report any code bugs to us via cuhkrlcourse@googlegroups.com or via github issue. Before you submission, remember to check the submit instruction at the directory of this assignment and make sure the required contents are included in your submission.\n",
    "\n",
    "Before you get start, remember to follow the instruction at https://github.com/cuhkrlcourse/ierg6130 to setup your environment.\n",
    "\n",
    "We will cover the following knowledege in this assignment:\n",
    "\n",
    "1. Basic policy gradient method\n",
    "2. Policy gradient with baseline\n",
    "3. Actor-critic framework\n",
    "\n",
    "You need to install some packages.\n",
    "1. Install `yaml` package via `pip install pyyaml` to print training information.\n",
    "2. Install `pandas` via `pip install pandas` to organize the learning progress.\n",
    "3. Install `matplotlib` via `pip install matplotlib` to visualize the learning result.\n",
    "4. Install `seaborn` via `pip install seaborn` to visualize the learning result (beautiful than matplotlib)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------\n",
    "\n",
    "## Section 1: Basic reinforcement learning pipeline\n",
    "\n",
    "(5 / 100 points)\n",
    "\n",
    "In this section, we will prepare several functions for evaulation, training RL algorithms. We will also build an `AbstractTrainer` class used as a general framework which left blanks for policy gradient methods.\n",
    "\n",
    "A essential difference of this assignment compared to the previous is that: the sampling and the optimization are splited into two phase. In each training iteration, the agent first collects a batch of samples followed by conducting the policy optimization. This modification allows us to have a clear view on the training process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "import time\n",
    "import gym\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import seaborn as sns\n",
    "import pandas as pd\n",
    "\n",
    "from utils import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "def run(trainer_cls, config=None, reward_threshold=None):\n",
    "    \"\"\"Run the trainer and report progress, agnostic to the class of trainer\n",
    "    :param trainer_cls: A trainer class \n",
    "    :param config: A dict\n",
    "    :param reward_threshold: the reward threshold to break the training\n",
    "    :return: The trained trainer and a dataframe containing learning progress\n",
    "    \"\"\"\n",
    "    assert inspect.isclass(trainer_cls)\n",
    "    if config is None:\n",
    "        config = {}\n",
    "    trainer = trainer_cls(config)\n",
    "    config = trainer.config\n",
    "    start = now = time.time()\n",
    "    stats = []\n",
    "    print(\"=== {} {} Training Start ===\".format(trainer_cls.__name__, config[\"env_name\"]))\n",
    "    pretty_print({\"Config\": config})\n",
    "    for i in range(config['max_iteration'] + 1):\n",
    "        stat = trainer.train()\n",
    "        stats.append(stat or {})\n",
    "        if i % config['evaluate_interval'] == 0 or \\\n",
    "                i == config[\"max_iteration\"]:\n",
    "            reward = trainer.evaluate(config[\"evaluate_num_episodes\"])\n",
    "            stat[\"evaluate_reward\"] = reward\n",
    "            print(\"=== {} {} Iteration {} ===\".format(\n",
    "                trainer_cls.__name__, config[\"env_name\"], i)\n",
    "            )\n",
    "            pretty_print({\"Training Progress\": stat})\n",
    "            now = time.time()\n",
    "        if reward_threshold is not None and reward > reward_threshold:\n",
    "            print(\"In {} iteration, current mean episode reward {:.3f} is \"\n",
    "                  \"greater than reward threshold {}. Congratulation! Now we \"\n",
    "                  \"exit the training process.\".format(\n",
    "                i, reward, reward_threshold))\n",
    "            break\n",
    "    stats = pd.DataFrame(stats)\n",
    "    return trainer, stats\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solve the TODOs and remove `pass`\n",
    "\n",
    "default_config = dict(\n",
    "    env_name=\"CartPole-v0\",\n",
    "    \n",
    "    max_iteration=1000,\n",
    "    max_episode_length=1000,\n",
    "    \n",
    "    train_batch_size=1000,\n",
    "    gamma=0.99,\n",
    "    learning_rate=0.01,\n",
    "    seed=0,\n",
    "    \n",
    "    evaluate_interval=10,\n",
    "    evaluate_num_episodes=50\n",
    ")\n",
    "\n",
    "\n",
    "class AbstractTrainer:\n",
    "    \"\"\"This is the abstract class for value-based RL trainer. We will inherent\n",
    "    the specify algorithm's trainer from this abstract class, so that we can\n",
    "    reuse the codes.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, config=None):\n",
    "        self.config = check_and_merge_config(config or {}, default_config)\n",
    "\n",
    "        # Create the environment\n",
    "        self.env_name = self.config['env_name']\n",
    "        self.env = gym.make(self.env_name)\n",
    "        if \"ram\" in self.env_name:  # Special process for Atari games\n",
    "            self.env = wrap_deepmind_ram(self.env)\n",
    "\n",
    "        # Apply the random seed\n",
    "        self.seed = self.config[\"seed\"]\n",
    "        seed_everything(self.seed)\n",
    "        self.env.seed(self.seed)\n",
    "\n",
    "        assert isinstance(self.env.observation_space,\n",
    "                          gym.spaces.Box), self.env.observation_space\n",
    "        self.obs_dim = self.env.observation_space.shape[0]\n",
    "        self.discrete_obs = True\n",
    "\n",
    "        assert isinstance(self.env.action_space, gym.spaces.discrete.Discrete)\n",
    "        self.act_dim = self.env.action_space.n\n",
    "\n",
    "        self.iteration = 0\n",
    "        self.start_time = time.time()\n",
    "        self.iteration_time = self.start_time\n",
    "        self.total_timesteps = 0\n",
    "        self.total_episodes = 0\n",
    "\n",
    "        # build the model\n",
    "        self.build_model()\n",
    "\n",
    "    def build_model(self):\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    # ===== Helper functions =====\n",
    "    def evaluate(self, num_episodes=10, render=False):\n",
    "        \"\"\"Evaluate the agents for num_episodes episodes.\"\"\"\n",
    "        return evaluate_agent(self, self.env, num_episodes, render)\n",
    "\n",
    "    def to_tensor(self, array):\n",
    "        \"\"\"Preprocess the observation.\"\"\"\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    def compute_action(self, observation):\n",
    "        \"\"\"Return the action, which can feed to self.env.step immediately.\"\"\"\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    def compute_values(self, observation):\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    def compute_log_probs(self, observation):\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    # ===== Training-related functions =====\n",
    "    def collect_samples(self):\n",
    "        \"\"\"Here we define the pipeline to collect sample even though\n",
    "        any specify functions are not implemented yet.\n",
    "        \"\"\"\n",
    "        iter_timesteps = 0\n",
    "        iter_episodes = 0\n",
    "        episode_lens = []\n",
    "        episode_rewards = []\n",
    "        episode_obs_list = []\n",
    "        episode_act_list = []\n",
    "        episode_reward_list = []\n",
    "        while iter_timesteps <= self.config[\"train_batch_size\"]:\n",
    "            obs_list, act_list, reward_list = [], [], []\n",
    "            obs = self.env.reset()\n",
    "            steps = 0\n",
    "            episode_reward = 0\n",
    "            while True:\n",
    "                act = self.compute_action(obs)\n",
    "                next_obs, reward, done, _ = self.env.step(act)\n",
    "\n",
    "                obs_list.append(obs)\n",
    "                act_list.append(act)\n",
    "                reward_list.append(reward)\n",
    "\n",
    "                obs = next_obs.copy()\n",
    "                steps += 1\n",
    "                episode_reward += reward\n",
    "                if done or steps > self.config[\"max_episode_length\"]:\n",
    "                    break\n",
    "            iter_timesteps += steps\n",
    "            iter_episodes += 1\n",
    "            episode_rewards.append(episode_reward)\n",
    "            episode_lens.append(steps)\n",
    "            episode_obs_list.append(np.array(obs_list, dtype=np.float32))\n",
    "            episode_act_list.append(np.array(act_list, dtype=np.float32))\n",
    "            episode_reward_list.append(np.array(reward_list, dtype=np.float32))\n",
    "        \n",
    "        # [TODO] Uncomment everything below and understand the data structure:\n",
    "        # The return `samples` is a dict that contains several fields.\n",
    "        # Each field (key-value pair) contains a list.\n",
    "        # Each element in list is a list represent the data in one trajectory (episode).\n",
    "        # Each episode list contains the data of that field of all time steps in that episode.\n",
    "        # The return `sample_info` is a dict contains logging item name and its value.\n",
    "        samples = {\n",
    "            \"obs\": episode_obs_list,\n",
    "            \"act\": episode_act_list,\n",
    "            \"reward\": episode_reward_list\n",
    "        }\n",
    "        \n",
    "        sample_info = {\n",
    "            \"iter_timesteps\": iter_timesteps,\n",
    "            \"iter_episodes\": iter_episodes,\n",
    "            \"performance\": np.mean(episode_rewards),  # help drawing figures\n",
    "            \"training_episode_length\": {\n",
    "                \"mean\": float(np.mean(episode_lens)),\n",
    "                \"max\": float(np.max(episode_lens)),\n",
    "                \"min\": float(np.min(episode_lens))\n",
    "            },\n",
    "            \"training_episode_reward\": {\n",
    "                \"mean\": float(np.mean(episode_rewards)),\n",
    "                \"max\": float(np.max(episode_rewards)),\n",
    "                \"min\": float(np.min(episode_rewards))\n",
    "            }\n",
    "        }\n",
    "        return samples, sample_info\n",
    "\n",
    "    def process_samples(self, samples):\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    def update_model(self, processed_samples):\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    # ===== Training iteration =====\n",
    "    def train(self):\n",
    "        \"\"\"Here we defined the training pipeline using the abstract\n",
    "        functions.\"\"\"\n",
    "        info = dict(iteration=self.iteration)\n",
    "\n",
    "        # [TODO] Uncomment the following block and go through the learning \n",
    "        # pipeline.\n",
    "        # Collect samples\n",
    "        samples, sample_info = self.collect_samples()\n",
    "        info.update(sample_info)\n",
    "\n",
    "        # Process samples\n",
    "        processed_samples, processed_info = self.process_samples(samples)\n",
    "        info.update(processed_info)\n",
    "\n",
    "        # Update the model\n",
    "        update_info = self.update_model(processed_samples)\n",
    "        info.update(update_info)\n",
    "\n",
    "        now = time.time()\n",
    "        self.iteration += 1\n",
    "        self.total_timesteps += info[\"iter_timesteps\"]\n",
    "        self.total_episodes += info[\"iter_episodes\"]\n",
    "        info[\"iter_time\"] = now - self.iteration_time\n",
    "        info[\"total_time\"] = now - self.start_time\n",
    "        info[\"total_episodes\"] = self.total_episodes\n",
    "        info[\"total_timesteps\"] = self.total_timesteps\n",
    "        self.iteration_time = now\n",
    "        return info\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Passed!\n"
     ]
    }
   ],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "class TestAbstractTrainer(AbstractTrainer):\n",
    "    def build_model(self):\n",
    "        pass\n",
    "    def compute_action(self, _):\n",
    "        return np.random.choice(self.act_dim)\n",
    "test_trainer = TestAbstractTrainer()\n",
    "s, s_info = test_trainer.collect_samples()\n",
    "assert len(s[\"obs\"]) == s_info[\"iter_episodes\"]\n",
    "for i in range(len(s[\"obs\"])):\n",
    "    assert s[\"obs\"][i].shape[0] <= default_config[\"max_episode_length\"]\n",
    "    assert len(s[\"act\"][i]) == len(s[\"obs\"][i]) == len(s[\"reward\"][i])\n",
    "assert abs(s_info[\"training_episode_length\"][\"mean\"] * s_info[\"iter_episodes\"]\n",
    "           - s_info[\"iter_timesteps\"]) < 1\n",
    "print(\"Test Passed!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have an abstract trainer class here. We have abstracted the learning process into three phases: Collect samples, process samples and update the model. In the following sections, we will continue to fulfill the blank functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 2: Basic policy gradient method\n",
    "\n",
    "(40 / 100 points)\n",
    "\n",
    "Unlike supervised learning, in RL setting the signal: reward and the output of neural network: the action distribution parameter (like the logits in discrete case or the mean in continous case) have no direct connection in terms of gradient. That is, you can't compute the gradient from the environment to your network output. \n",
    "\n",
    "To leverage the powerful tool of gradient descent, scientists innovated a technique called policy gradient, which is computed on top of the rewards and policy network's output. Scientists prove that: the policy gradient is the gradient of expected return w.r.t. the output. So that by applying gradient descent to gradient descent algorithm you can optimize your policy network and maximize the expected return (or minimize the negative of expected return).\n",
    "\n",
    "Concretely, if we consider the expected return, the value we want to maximize, is:\n",
    "\n",
    "$$Q = \\mathbb E_{\\text{possible trajectories}} \\sum_t r(a_t, s_t) = \\sum_{s_0, a_0,..} p(s_0, a_0, ..., s_t, a_t) r(s_0, a_0, ..., s_t, a_t) = \\sum_{\\tau} p(\\tau)r(\\tau)$$ \n",
    "\n",
    "wherein $\\sum_t r(a_t, s_t) = r(s_0, a_0, ..., s_t, a_t) = r(\\tau)$ is the trajectory return. We use $\\tau$ to represent a trajectory $s_0, a_0, ..., s_t, a_t$. Note that we remove the discount factor for simplicity.\n",
    "\n",
    "Since we want to maximize Q, we can simply compute the gradient from Q to parameter $\\theta$ (which is included in the action probability $p(a_t)$):\n",
    "\n",
    "$$\\nabla_\\theta Q = \\nabla_\\theta \\sum_{\\tau} p(\\tau)r(\\tau) = \\sum_{\\tau} r(\\tau) \\nabla_\\theta p(\\tau)$$\n",
    "\n",
    "Scientists use a famous trick to deal with the gradient of probability of trajectory: $\\nabla_\\theta p(\\tau) = p(\\tau)\\cfrac{\\nabla_\\theta p(\\tau)}{p(\\tau)} = p(\\tau)\\nabla_\\theta \\log p(\\tau)$. \n",
    "\n",
    "The reason for this trick is that the probability of a trajectory is the product of all related probabilities of states and actions. So introducing a log term can change the production to summation: $p_\\theta(\\tau) = p(s_0, a_0, ...) = p(s_0) \\prod_t \\pi_\\theta (a_t|s_t) p(s_{t+1}|s_t, a_t)$ Now we can expand the log of product above to sum of log:\n",
    "\n",
    "$$\\log p_\\theta (\\tau) = \\log p(s_1) + \\sum_t \\log \\pi_\\theta(a_t|s_t) + \\sum_t \\log p(s_{t+1}|s_t, a_t)$$\n",
    "\n",
    "You will find that the first and third term are not related to the parameter of policy $\\pi_\\theta(\\cdot)$. So when we back to $\\nabla_\\theta Q$, we find \n",
    "\n",
    "$$\\nabla_\\theta Q = \\sum_{\\tau} r(\\tau) \\nabla_\\theta p(\\tau) =  \\sum p_\\theta(\\tau) ( \\sum_t  \\nabla_\\theta \\log \\pi_\\theta(a_t|s_t) ) r(\\tau) d\\tau$$\n",
    "\n",
    "The probability of trajectory is included when we sampled sufficient data from environment (Yes, so REINFORCE is a Monte Carlo algorithm) so the final form of policy gradient is:\n",
    "\n",
    "$$\\nabla_\\theta Q =\\cfrac{1}{N}\\sum_{i=1}^N [( \\sum_t  \\nabla_\\theta \\log \\pi_\\theta(a_{i,t}|s_{i,t}) (\\sum_{t'=t} \\gamma^{t'-t} r(s_{i,t'}, a_{i,t'}) )]$$\n",
    "\n",
    "This algorithm is called REINFORCE algorithm, which is a Monte Carlo Policy Gradient algorithm having long history. In this section, we will implement the it using pytorch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The policy network is composed by two parts: a basic MLP serve as function approximator which predict the probabilities of actions given the observation; and a distribution layer building upon the MLP to wrap the raw logits output of neural network to a distribution and provides API for action sampling and log probability retrieving."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self, obs_dim, act_dim, hidden_units=128):\n",
    "        super(Net, self).__init__()\n",
    "        self.fc0 = nn.Linear(obs_dim, hidden_units)\n",
    "        self.fc1 = nn.Linear(hidden_units, act_dim)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x.type_as(self.fc0.bias)\n",
    "        x = F.relu(self.fc0(x))\n",
    "        x = self.fc1(x)\n",
    "        return x\n",
    "\n",
    "\n",
    "class CategoricalPolicy(nn.Module):\n",
    "    def __init__(self, obs_dim, act_dim, hidden_units=128):\n",
    "        super(CategoricalPolicy, self).__init__()\n",
    "        self.network = Net(obs_dim, act_dim, hidden_units)\n",
    "\n",
    "    def forward(self, obs):\n",
    "        logit = self.network(obs)\n",
    "        \n",
    "        # [TODO] Create an object of the class \"Categorical\" in torch \n",
    "        # package \"distributions\". Then sample an action from it.\n",
    "        # Creates a categorical distribution parameterized by either :attr:`probs` or\n",
    "        # :attr:`logits` (but not both).\n",
    "        logit = F.softmax(logit,dim=-1)\n",
    "        action = torch.distributions.Categorical(logit).sample().item()\n",
    "        \n",
    "        return action\n",
    "\n",
    "    def log_prob(self, obs, act):\n",
    "        logits = self.network(obs)\n",
    "        \n",
    "        # [TODO] Create an object of the class \"Categorical\" in torch \n",
    "        # package \"distributions\". Then collect the log probability of\n",
    "        # the action in this distribution.\n",
    "        logits = F.softmax(logits,dim=-1)\n",
    "        log_prob = torch.distributions.Categorical(logits).log_prob(act)\n",
    "        \n",
    "        return log_prob\n",
    "\n",
    "# Not that we do not implement GaussianPolicy here. So we can't\n",
    "# apply our algorithm to the environment with continous action.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solve the TODOs and remove `pass`\n",
    "\n",
    "pg_default_config = merge_config(dict(\n",
    "    normalize_advantage=True,\n",
    "    clip_gradient=10.0,\n",
    "    hidden_units=64,\n",
    "), default_config)\n",
    "\n",
    "\n",
    "class PGTrainer(AbstractTrainer):\n",
    "    def __init__(self, config=None):\n",
    "        config = check_and_merge_config(config or {}, pg_default_config)\n",
    "        super().__init__(config)\n",
    "\n",
    "    def build_model(self):\n",
    "        \"\"\"Build the policy network and related optimizer\"\"\"\n",
    "        # Detect whether you have GPU or not. Remember to call X.to(self.device)\n",
    "        # if necessary.\n",
    "        self.device = torch.device(\n",
    "            \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "        )\n",
    "        \n",
    "        # [TODO] Build the policy network using CategoricalPolicy\n",
    "        # Hint: Remember to pass config[\"hidden_units\"], and set policy network\n",
    "        #  to the device you are using.\n",
    "        self.policy = CategoricalPolicy(self.obs_dim, self.act_dim, self.config[\"hidden_units\"]).to(self.device)\n",
    "        \n",
    "        # Build the Adam optimizer.\n",
    "        self.optimizer = torch.optim.Adam(\n",
    "            self.policy.parameters(),\n",
    "            lr=self.config[\"learning_rate\"]\n",
    "        )\n",
    "\n",
    "    def to_tensor(self, array):\n",
    "        \"\"\"Transform a numpy array to a pytorch tensor\"\"\"\n",
    "        return torch.from_numpy(array).to(self.device)\n",
    "\n",
    "    def to_array(self, tensor):\n",
    "        \"\"\"Transform a pytorch tensor to a numpy array\"\"\"\n",
    "        return tensor.cpu().detach().numpy()\n",
    "\n",
    "    def compute_action(self, observation):\n",
    "        \"\"\"Compute the action for single observation\"\"\"\n",
    "        assert observation.ndim == 1\n",
    "        # [TODO] Sample an action from action distribution given by the policy\n",
    "        # Hint: The input of policy network is a batch of data, so you need to\n",
    "        #  expand the first dimension of observation before feeding it to policy network.\n",
    "        observation = torch.from_numpy(observation).unsqueeze(0)\n",
    "        action = self.policy.forward(observation)\n",
    "        return action\n",
    "\n",
    "    def compute_log_probs(self, observation, action):\n",
    "        \"\"\"Compute the log probabilities of a batch of state-action pair\"\"\"\n",
    "        # [TODO] Using the function of policy network to get log probs.\n",
    "        # Hint: Remember to transform the data into tensor before feeding it.\n",
    "        observation = torch.from_numpy(observation).unsqueeze(0).to(self.device)\n",
    "        action = torch.from_numpy(action).to(self.device)\n",
    "        \n",
    "        log_probs = self.policy.log_prob(observation, action).squeeze().to(self.device)\n",
    "        return log_probs\n",
    "\n",
    "    def process_samples(self, samples):\n",
    "        \"\"\"Process samples and add advantages in it\"\"\"\n",
    "        values = []\n",
    "        for reward_list in samples[\"reward\"]:\n",
    "            # reward_list contains rewards in one episode\n",
    "            returns = np.zeros_like(reward_list, dtype=np.float32)\n",
    "            Q = 0\n",
    "            \n",
    "            # [TODO] Scan the episode in a reverse order and compute the\n",
    "            # discounted return at each time step. Fill the array `returns`\n",
    "            # Hint: returns[n-1] = r_n-1\n",
    "            #       returns[n-2] = r_n-2 + gamma * r_n-1\n",
    "            #       ...\n",
    "            #       returns[0] = r_0 + gamma * r_1 + gamma^2 * r_2 + ...\n",
    "            #  wherein n is len(reward_list)\n",
    "            n = len(reward_list)\n",
    "            for i in reversed(range(n)):\n",
    "                if i == n-1:\n",
    "                    returns[i] = reward_list[i]\n",
    "                else:\n",
    "                    returns[i] = reward_list[i] + self.config['gamma'] * returns[i+1]\n",
    "                \n",
    "            values.append(returns)\n",
    "\n",
    "        # We call the values advantage here.\n",
    "        advantages = np.concatenate(values)\n",
    "\n",
    "        if self.config[\"normalize_advantage\"]:\n",
    "            # [TODO] normalize the advantage so that it's mean is\n",
    "            # almost 0 and the its standard deviation is almost 1.\n",
    "            advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-15)\n",
    "        \n",
    "        samples[\"advantages\"] = advantages\n",
    "        return samples, {}\n",
    "\n",
    "    def update_model(self, processed_samples):\n",
    "        \"\"\"A wrapper function, call self.update_policy\"\"\"\n",
    "        return self.update_policy(processed_samples)\n",
    "\n",
    "    def update_policy(self, processed_samples):\n",
    "        \"\"\"Update the policy network\"\"\"\n",
    "        advantages = self.to_tensor(processed_samples[\"advantages\"])\n",
    "        flat_obs = np.concatenate(processed_samples[\"obs\"])\n",
    "        flat_act = np.concatenate(processed_samples[\"act\"])\n",
    "\n",
    "        self.policy.train()\n",
    "        self.optimizer.zero_grad()\n",
    "\n",
    "        log_probs = self.compute_log_probs(flat_obs, flat_act)\n",
    "\n",
    "        assert log_probs.shape == advantages.shape, \"log_probs shape {} is not \" \\\n",
    "            \"compatible with advantages {}\".format(log_probs.shape, advantages.shape)\n",
    "        \n",
    "        # [TODO] Compute the loss using log probabilities and advantages.\n",
    "        # The minus here is importance!\n",
    "        loss =  - (log_probs * advantages).sum()\n",
    "        \n",
    "        loss.backward()\n",
    "        \n",
    "        # Clip the gradient\n",
    "        torch.nn.utils.clip_grad_norm_(\n",
    "            self.policy.parameters(), self.config[\"clip_gradient\"]\n",
    "        )\n",
    "\n",
    "        self.optimizer.step()\n",
    "        self.policy.eval()\n",
    "        update_info = {\n",
    "            \"policy_loss\": loss.item(),\n",
    "            \"mean_log_prob\": torch.mean(log_probs).item(),\n",
    "            \"mean_advantage\": torch.mean(advantages).item()\n",
    "        }\n",
    "        return update_info\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Passed!\n"
     ]
    }
   ],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "# Test advantage computing\n",
    "test_trainer = PGTrainer({\"normalize_advantage\": False})\n",
    "test_trainer.train()\n",
    "fake_sample = {\"reward\": [[2, 2, 2, 2, 2]]}\n",
    "assert_almost_equal(\n",
    "    test_trainer.process_samples(fake_sample)[0][\"reward\"][0],\n",
    "    fake_sample[\"reward\"][0]\n",
    ")\n",
    "assert_almost_equal(\n",
    "    test_trainer.process_samples(fake_sample)[0][\"advantages\"],\n",
    "    np.array([9.80199, 7.880798, 5.9402, 3.98, 2.], dtype=np.float32)\n",
    ")\n",
    "\n",
    "# Test advantage normalization\n",
    "test_trainer = PGTrainer(\n",
    "    {\"normalize_advantage\": True, \"env_name\": \"LunarLander-v2\"})\n",
    "test_adv = test_trainer.process_samples(fake_sample)[0][\"advantages\"]\n",
    "assert_almost_equal(test_adv.mean(), 0.0)\n",
    "assert_almost_equal(test_adv.std(), 1.0)\n",
    "\n",
    "# Test the shape of functions' returns\n",
    "fake_observation = np.array([\n",
    "    test_trainer.env.observation_space.sample() for i in range(10)\n",
    "])\n",
    "fake_action = np.array([\n",
    "    test_trainer.env.action_space.sample() for i in range(10)\n",
    "])\n",
    "assert test_trainer.to_tensor(fake_observation).shape == torch.Size([10, 8])\n",
    "assert np.array(test_trainer.compute_action(fake_observation[0])).shape == ()\n",
    "assert test_trainer.compute_log_probs(fake_observation, fake_action).shape == \\\n",
    "       torch.Size([10])\n",
    "\n",
    "print(\"Test Passed!\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== PGTrainer CartPole-v0 Training Start ===\n",
      "Config:\n",
      "  checked: true\n",
      "  clip_gradient: 10.0\n",
      "  env_name: CartPole-v0\n",
      "  evaluate_interval: 10\n",
      "  evaluate_num_episodes: 50\n",
      "  gamma: 0.99\n",
      "  hidden_units: 64\n",
      "  learning_rate: 0.01\n",
      "  max_episode_length: 200\n",
      "  max_iteration: 1000\n",
      "  normalize_advantage: false\n",
      "  seed: 0\n",
      "  train_batch_size: 200\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 0 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 22.0\n",
      "  iter_episodes: 14\n",
      "  iter_time: 0.17351126670837402\n",
      "  iter_timesteps: 209\n",
      "  iteration: 0\n",
      "  mean_advantage: 8.392674446105957\n",
      "  mean_log_prob: -0.6853688359260559\n",
      "  performance: 14.928571428571429\n",
      "  policy_loss: 1216.459228515625\n",
      "  total_episodes: 14\n",
      "  total_time: 0.17351126670837402\n",
      "  total_timesteps: 209\n",
      "  training_episode_length:\n",
      "    max: 29.0\n",
      "    mean: 14.928571428571429\n",
      "    min: 10.0\n",
      "  training_episode_reward:\n",
      "    max: 29.0\n",
      "    mean: 14.928571428571429\n",
      "    min: 10.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 10 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 48.2\n",
      "  iter_episodes: 8\n",
      "  iter_time: 0.17852330207824707\n",
      "  iter_timesteps: 253\n",
      "  iteration: 10\n",
      "  mean_advantage: 16.79143714904785\n",
      "  mean_log_prob: -0.6800906658172607\n",
      "  performance: 31.625\n",
      "  policy_loss: 2771.17333984375\n",
      "  total_episodes: 92\n",
      "  total_time: 2.6159815788269043\n",
      "  total_timesteps: 2510\n",
      "  training_episode_length:\n",
      "    max: 54.0\n",
      "    mean: 31.625\n",
      "    min: 14.0\n",
      "  training_episode_reward:\n",
      "    max: 54.0\n",
      "    mean: 31.625\n",
      "    min: 14.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 20 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 73.62\n",
      "  iter_episodes: 5\n",
      "  iter_time: 0.17453265190124512\n",
      "  iter_timesteps: 247\n",
      "  iteration: 20\n",
      "  mean_advantage: 22.18447494506836\n",
      "  mean_log_prob: -0.5995378494262695\n",
      "  performance: 49.4\n",
      "  policy_loss: 3186.316162109375\n",
      "  total_episodes: 137\n",
      "  total_time: 6.220342397689819\n",
      "  total_timesteps: 4970\n",
      "  training_episode_length:\n",
      "    max: 62.0\n",
      "    mean: 49.4\n",
      "    min: 32.0\n",
      "  training_episode_reward:\n",
      "    max: 62.0\n",
      "    mean: 49.4\n",
      "    min: 32.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 30 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 84.96\n",
      "  iter_episodes: 3\n",
      "  iter_time: 0.16156840324401855\n",
      "  iter_timesteps: 215\n",
      "  iteration: 30\n",
      "  mean_advantage: 31.012895584106445\n",
      "  mean_log_prob: -0.5466681718826294\n",
      "  performance: 71.66666666666667\n",
      "  policy_loss: 3605.765869140625\n",
      "  total_episodes: 165\n",
      "  total_time: 10.67941927909851\n",
      "  total_timesteps: 7257\n",
      "  training_episode_length:\n",
      "    max: 90.0\n",
      "    mean: 71.66666666666667\n",
      "    min: 39.0\n",
      "  training_episode_reward:\n",
      "    max: 90.0\n",
      "    mean: 71.66666666666667\n",
      "    min: 39.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 40 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 50.96\n",
      "  iter_episodes: 5\n",
      "  iter_time: 0.21043729782104492\n",
      "  iter_timesteps: 266\n",
      "  iteration: 40\n",
      "  mean_advantage: 24.109878540039062\n",
      "  mean_log_prob: -0.5016810297966003\n",
      "  performance: 53.2\n",
      "  policy_loss: 3370.681640625\n",
      "  total_episodes: 204\n",
      "  total_time: 15.434703588485718\n",
      "  total_timesteps: 9717\n",
      "  training_episode_length:\n",
      "    max: 74.0\n",
      "    mean: 53.2\n",
      "    min: 34.0\n",
      "  training_episode_reward:\n",
      "    max: 74.0\n",
      "    mean: 53.2\n",
      "    min: 34.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 50 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 165.7\n",
      "  iter_episodes: 2\n",
      "  iter_time: 0.151594877243042\n",
      "  iter_timesteps: 205\n",
      "  iteration: 50\n",
      "  mean_advantage: 41.396663665771484\n",
      "  mean_log_prob: -0.5165351033210754\n",
      "  performance: 102.5\n",
      "  policy_loss: 4290.62548828125\n",
      "  total_episodes: 236\n",
      "  total_time: 19.076964616775513\n",
      "  total_timesteps: 12068\n",
      "  training_episode_length:\n",
      "    max: 147.0\n",
      "    mean: 102.5\n",
      "    min: 58.0\n",
      "  training_episode_reward:\n",
      "    max: 147.0\n",
      "    mean: 102.5\n",
      "    min: 58.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 60 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 174.04\n",
      "  iter_episodes: 2\n",
      "  iter_time: 0.2892270088195801\n",
      "  iter_timesteps: 400\n",
      "  iteration: 60\n",
      "  mean_advantage: 57.13198471069336\n",
      "  mean_log_prob: -0.5640437006950378\n",
      "  performance: 200.0\n",
      "  policy_loss: 12956.734375\n",
      "  total_episodes: 256\n",
      "  total_time: 27.38076138496399\n",
      "  total_timesteps: 15694\n",
      "  training_episode_length:\n",
      "    max: 200.0\n",
      "    mean: 200.0\n",
      "    min: 200.0\n",
      "  training_episode_reward:\n",
      "    max: 200.0\n",
      "    mean: 200.0\n",
      "    min: 200.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 70 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 193.12\n",
      "  iter_episodes: 2\n",
      "  iter_time: 0.27526402473449707\n",
      "  iter_timesteps: 400\n",
      "  iteration: 70\n",
      "  mean_advantage: 57.13198471069336\n",
      "  mean_log_prob: -0.5780906677246094\n",
      "  performance: 200.0\n",
      "  policy_loss: 13385.3828125\n",
      "  total_episodes: 276\n",
      "  total_time: 36.35875391960144\n",
      "  total_timesteps: 19371\n",
      "  training_episode_length:\n",
      "    max: 200.0\n",
      "    mean: 200.0\n",
      "    min: 200.0\n",
      "  training_episode_reward:\n",
      "    max: 200.0\n",
      "    mean: 200.0\n",
      "    min: 200.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 80 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 171.86\n",
      "  iter_episodes: 2\n",
      "  iter_time: 0.2762589454650879\n",
      "  iter_timesteps: 400\n",
      "  iteration: 80\n",
      "  mean_advantage: 57.13198471069336\n",
      "  mean_log_prob: -0.5759513974189758\n",
      "  performance: 200.0\n",
      "  policy_loss: 13180.2412109375\n",
      "  total_episodes: 296\n",
      "  total_time: 45.564138412475586\n",
      "  total_timesteps: 23193\n",
      "  training_episode_length:\n",
      "    max: 200.0\n",
      "    mean: 200.0\n",
      "    min: 200.0\n",
      "  training_episode_reward:\n",
      "    max: 200.0\n",
      "    mean: 200.0\n",
      "    min: 200.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 90 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 186.1\n",
      "  iter_episodes: 2\n",
      "  iter_time: 0.2563135623931885\n",
      "  iter_timesteps: 376\n",
      "  iteration: 90\n",
      "  mean_advantage: 55.3580207824707\n",
      "  mean_log_prob: -0.5797972679138184\n",
      "  performance: 188.0\n",
      "  policy_loss: 12123.35546875\n",
      "  total_episodes: 316\n",
      "  total_time: 53.94672465324402\n",
      "  total_timesteps: 26815\n",
      "  training_episode_length:\n",
      "    max: 200.0\n",
      "    mean: 188.0\n",
      "    min: 176.0\n",
      "  training_episode_reward:\n",
      "    max: 200.0\n",
      "    mean: 188.0\n",
      "    min: 176.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 100 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 150.3\n",
      "  iter_episodes: 2\n",
      "  iter_time: 0.22440004348754883\n",
      "  iter_timesteps: 334\n",
      "  iteration: 100\n",
      "  mean_advantage: 51.966278076171875\n",
      "  mean_log_prob: -0.5650166273117065\n",
      "  performance: 167.0\n",
      "  policy_loss: 9959.953125\n",
      "  total_episodes: 336\n",
      "  total_time: 62.53675580024719\n",
      "  total_timesteps: 30104\n",
      "  training_episode_length:\n",
      "    max: 185.0\n",
      "    mean: 167.0\n",
      "    min: 149.0\n",
      "  training_episode_reward:\n",
      "    max: 185.0\n",
      "    mean: 167.0\n",
      "    min: 149.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 110 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 196.48\n",
      "  iter_episodes: 2\n",
      "  iter_time: 0.27426648139953613\n",
      "  iter_timesteps: 400\n",
      "  iteration: 110\n",
      "  mean_advantage: 57.13198471069336\n",
      "  mean_log_prob: -0.5765231847763062\n",
      "  performance: 200.0\n",
      "  policy_loss: 12943.462890625\n",
      "  total_episodes: 356\n",
      "  total_time: 70.18829560279846\n",
      "  total_timesteps: 33613\n",
      "  training_episode_length:\n",
      "    max: 200.0\n",
      "    mean: 200.0\n",
      "    min: 200.0\n",
      "  training_episode_reward:\n",
      "    max: 200.0\n",
      "    mean: 200.0\n",
      "    min: 200.0\n",
      "\n",
      "In 110 iteration, current mean episode reward 196.480 is greater than reward threshold 195.0. Congratulation! Now we exit the training process.\n"
     ]
    }
   ],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "pg_trainer_no_na, pg_result_no_na = run(PGTrainer, dict(\n",
    "    learning_rate=0.01,\n",
    "    max_episode_length=200,\n",
    "    train_batch_size=200,\n",
    "    normalize_advantage=False,  # <<== Here!\n",
    "), 195.0)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== PGTrainer CartPole-v0 Training Start ===\n",
      "Config:\n",
      "  checked: true\n",
      "  clip_gradient: 10.0\n",
      "  env_name: CartPole-v0\n",
      "  evaluate_interval: 10\n",
      "  evaluate_num_episodes: 50\n",
      "  gamma: 0.99\n",
      "  hidden_units: 64\n",
      "  learning_rate: 0.01\n",
      "  max_episode_length: 200\n",
      "  max_iteration: 1000\n",
      "  normalize_advantage: true\n",
      "  seed: 0\n",
      "  train_batch_size: 200\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 0 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 27.88\n",
      "  iter_episodes: 14\n",
      "  iter_time: 0.17253828048706055\n",
      "  iter_timesteps: 209\n",
      "  iteration: 0\n",
      "  mean_advantage: 1.7339532121241064e-07\n",
      "  mean_log_prob: -0.6853688359260559\n",
      "  performance: 14.928571428571429\n",
      "  policy_loss: 2.657162666320801\n",
      "  total_episodes: 14\n",
      "  total_time: 0.17253828048706055\n",
      "  total_timesteps: 209\n",
      "  training_episode_length:\n",
      "    max: 29.0\n",
      "    mean: 14.928571428571429\n",
      "    min: 10.0\n",
      "  training_episode_reward:\n",
      "    max: 29.0\n",
      "    mean: 14.928571428571429\n",
      "    min: 10.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 10 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 53.86\n",
      "  iter_episodes: 4\n",
      "  iter_time: 0.1625657081604004\n",
      "  iter_timesteps: 240\n",
      "  iteration: 10\n",
      "  mean_advantage: -1.0728836485895954e-07\n",
      "  mean_log_prob: -0.6183241605758667\n",
      "  performance: 60.0\n",
      "  policy_loss: 6.967897415161133\n",
      "  total_episodes: 80\n",
      "  total_time: 2.711747646331787\n",
      "  total_timesteps: 2464\n",
      "  training_episode_length:\n",
      "    max: 86.0\n",
      "    mean: 60.0\n",
      "    min: 33.0\n",
      "  training_episode_reward:\n",
      "    max: 86.0\n",
      "    mean: 60.0\n",
      "    min: 33.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 20 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 141.92\n",
      "  iter_episodes: 2\n",
      "  iter_time: 0.20844125747680664\n",
      "  iter_timesteps: 301\n",
      "  iteration: 20\n",
      "  mean_advantage: -7.604047880249709e-08\n",
      "  mean_log_prob: -0.5834347605705261\n",
      "  performance: 150.5\n",
      "  policy_loss: -9.724409103393555\n",
      "  total_episodes: 113\n",
      "  total_time: 6.4148454666137695\n",
      "  total_timesteps: 5110\n",
      "  training_episode_length:\n",
      "    max: 200.0\n",
      "    mean: 150.5\n",
      "    min: 101.0\n",
      "  training_episode_reward:\n",
      "    max: 200.0\n",
      "    mean: 150.5\n",
      "    min: 101.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 30 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 178.36\n",
      "  iter_episodes: 2\n",
      "  iter_time: 0.27127528190612793\n",
      "  iter_timesteps: 384\n",
      "  iteration: 30\n",
      "  mean_advantage: -8.443991816875496e-08\n",
      "  mean_log_prob: -0.5592540502548218\n",
      "  performance: 192.0\n",
      "  policy_loss: -8.075216293334961\n",
      "  total_episodes: 134\n",
      "  total_time: 13.088002681732178\n",
      "  total_timesteps: 8017\n",
      "  training_episode_length:\n",
      "    max: 200.0\n",
      "    mean: 192.0\n",
      "    min: 184.0\n",
      "  training_episode_reward:\n",
      "    max: 200.0\n",
      "    mean: 192.0\n",
      "    min: 184.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 40 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 185.92\n",
      "  iter_episodes: 2\n",
      "  iter_time: 0.2732682228088379\n",
      "  iter_timesteps: 400\n",
      "  iteration: 40\n",
      "  mean_advantage: -1.335144048653092e-07\n",
      "  mean_log_prob: -0.5304846167564392\n",
      "  performance: 200.0\n",
      "  policy_loss: -2.1693427562713623\n",
      "  total_episodes: 154\n",
      "  total_time: 21.900437831878662\n",
      "  total_timesteps: 11945\n",
      "  training_episode_length:\n",
      "    max: 200.0\n",
      "    mean: 200.0\n",
      "    min: 200.0\n",
      "  training_episode_reward:\n",
      "    max: 200.0\n",
      "    mean: 200.0\n",
      "    min: 200.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 50 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 127.22\n",
      "  iter_episodes: 2\n",
      "  iter_time: 0.20245862007141113\n",
      "  iter_timesteps: 303\n",
      "  iteration: 50\n",
      "  mean_advantage: 1.8884641406202718e-08\n",
      "  mean_log_prob: -0.5241126418113708\n",
      "  performance: 151.5\n",
      "  policy_loss: -0.31362247467041016\n",
      "  total_episodes: 174\n",
      "  total_time: 30.298981428146362\n",
      "  total_timesteps: 15126\n",
      "  training_episode_length:\n",
      "    max: 161.0\n",
      "    mean: 151.5\n",
      "    min: 142.0\n",
      "  training_episode_reward:\n",
      "    max: 161.0\n",
      "    mean: 151.5\n",
      "    min: 142.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 60 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 131.02\n",
      "  iter_episodes: 2\n",
      "  iter_time: 0.2373652458190918\n",
      "  iter_timesteps: 329\n",
      "  iteration: 60\n",
      "  mean_advantage: -2.5943418791030126e-07\n",
      "  mean_log_prob: -0.5278941988945007\n",
      "  performance: 164.5\n",
      "  policy_loss: -2.1950151920318604\n",
      "  total_episodes: 196\n",
      "  total_time: 36.40465545654297\n",
      "  total_timesteps: 17858\n",
      "  training_episode_length:\n",
      "    max: 190.0\n",
      "    mean: 164.5\n",
      "    min: 139.0\n",
      "  training_episode_reward:\n",
      "    max: 190.0\n",
      "    mean: 164.5\n",
      "    min: 139.0\n",
      "\n",
      "=== PGTrainer CartPole-v0 Iteration 70 ===\n",
      "Training Progress:\n",
      "  evaluate_reward: 200.0\n",
      "  iter_episodes: 2\n",
      "  iter_time: 0.2682826519012451\n",
      "  iter_timesteps: 400\n",
      "  iteration: 70\n",
      "  mean_advantage: -1.335144048653092e-07\n",
      "  mean_log_prob: -0.4820900559425354\n",
      "  performance: 200.0\n",
      "  policy_loss: -6.846751689910889\n",
      "  total_episodes: 216\n",
      "  total_time: 43.1685688495636\n",
      "  total_timesteps: 21228\n",
      "  training_episode_length:\n",
      "    max: 200.0\n",
      "    mean: 200.0\n",
      "    min: 200.0\n",
      "  training_episode_reward:\n",
      "    max: 200.0\n",
      "    mean: 200.0\n",
      "    min: 200.0\n",
      "\n",
      "In 70 iteration, current mean episode reward 200.000 is greater than reward threshold 195.0. Congratulation! Now we exit the training process.\n"
     ]
    }
   ],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "pg_trainer_na, pg_result_na = run(PGTrainer, dict(\n",
    "    learning_rate=0.01,\n",
    "    max_episode_length=200,\n",
    "    train_batch_size=200,\n",
    "    normalize_advantage=True,  # <<== Here!\n",
    "), 195.0)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Policy Gradient')"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEXCAYAAACH/8KRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsnXd8ZGd97r+/6TMalZW0q9U27669LrvuGDDN2NgE4xgvJbSEEkqACzfAJbkhIfcGQsJN7g2kQEJxAgECMWCIKSGAbbDBBmxwb+ut9vZVXZWRps97/3jPO+dMPyNppJF0ns9nPyOdOefMOzPa85zn+TVRSuHBgwcPHjzMFb6lXoAHDx48eFje8IjEgwcPHjzMCx6RePDgwYOHecEjEg8ePHjwMC94ROLBgwcPHuYFj0g8ePDgwcO84BGJhxUNEfmIiHzF+nmLiCRExL/U62oWInKliBxz/P64iFy5hEvy4KEIj0g8LAuIyNMikrSIYEhE/lVE4s2cQyl1RCkVV0rlF3htnSLyt9YaZ0TkiIh8U0SetZCv44RSapdS6s75nsdJtB48zBUekXhYTniZUioOXAo8E/hfS7weRCQM/AS4ALge6ALOA74GXFfjmMCiLdCDh0WARyQelh2UUseBHwDnA4jIBhH5roiMi8gBEfm9aseJyFYRUeZCLiK9lrI5ISKnReTb1vbHRORljuOCIjIqIhdXOe0bgU3Ay5VSjyml8kqpGaXUN5VSH3GcQ4nIe0RkP7Df2vYPInJURKZE5H4ReYFj/6iIfNFa1xNo4nS+l6dF5BrrZ5+I/LGIHBSRMRH5hoj0lr3nN1tKaVRE/tR67lrgQ8BrLaX3cDPfgwcPBh6ReFh2EJHN6Lv9B61NNwHHgA3AbwH/R0SudnGqfwNiwC5gHfB31vYvA29w7HcdcFIp9VCVc1wD/EgpNePi9V4OPBvYaf3+a+BioBf4d+BmEYlYz30YONP69xLgzXXO+17r3C9EfwangX8q2+f5wDnA1cCfich5SqkfAv8H+Lpl+V3k4j148FABj0g8LCd8W0QmgLuBn6IJYzP6IvlBpVTKutj/C1op1ISIDAIvBd6llDqtlMoqpX5qPf0V4DoR6bJ+fyOadKqhHzjlOO/FIjJhqYy9Zfv+lVJqXCmVBFBKfUUpNaaUyimlPgGE0Rd7gNcAH7P2Pwp8ss7beSfwp0qpY0qpNPAR4LfKLLQ/V0ollVIPAw8DHml4WDB4ROJhOeHlSqkepdQZSql3WxfkDcC4Umrasd9hYGODc222jjtd/oRS6gTwc+BVItKDJpyv1jjPGDDoOPYhpVQP8Eo0MThx1PmLiPyBiOwRkUmLILvRxIT1vpz7H67zXs4AbrEIbALYA+SBAcc+pxw/zwJNJSp48FAPHpF4WO44AfSKSKdj2xbgeIPjjlrH9dR4/ktoe+vVwC+tuEw1/Bj4DRHpcLHWYqttKx7yQbTyWGORzyQg1i4n0WRnsKXOeY8CL7VI1vyL1Flz1TV58DBXeETiYVnDsn1+AfyViERE5ELgbdRWEOa4k+iA/adFZI0VUL/Cscu30dlh70PHTGrhy+iL/i0icr6I+K04x2UNlt4J5IARICAif4bO+DL4BvAn1to2Ab9f51yfBT4mImcAiMhaEdnd4PUNhoCtIuJdCzzMGd4fj4eVgNcDW9Hq5Bbgw0qp21wc90YgCzwJDAPvN09Yttm3gG3Af9Q6gVIqBVwFPAF8H5gC9qKzrF5T57V/hCayfWjbKkWplfXn1vangFupHaMB+Afgu8CtIjIN3IMO6rvBzdbjmIg84PIYDx5KIN5gKw8eqsNSCWcrpd7QcGcPHlYxvMIoDx6qwKrDeBsNsr88ePDgWVsePFTAKmg8CvxAKfWzpV6PBw/tDs/a8uDBgwcP84KnSDx48ODBw7ywrGMk/f39auvWrUu9DA8ePHhYVrj//vtHlVJrF+p8y5pItm7dyn333bfUy/DgwYOHZQURqdcpoWl41pYHDx48eJgXPCLx4MGDBw/zgkckHjx48OBhXvCIxIMHDx48zAsekXjw4MGDh3mhZUQiIptF5A5r3sLjIvI+a3uviNwmIvutxzXWdhGRT1qjUh8RkUtbtTYPHjx48LBwaKUiyQF/oJQ6D7gceI+I7AT+GPixUmoHepbDH1v7vxTYYf17B/CZFq7NgwcPHjwsEFpWR2LNezhp/TwtInvQU+t2A1dau30JuBM94Gc38GWle7bcIyI9IjJoncfDUiE5AY/fAoWcvW3rC2DduZX77r8dTj9l/x5dA+e/CkRK9yvk4dGbIT2NK4Q74YJXg8/f/PqVIrf3h3wncT6z2Xxxs6+QZWD815zsf27FId2xEC+7cBApW/fd+0e5bOsaIsHq60hm8tzz1BhXnr224thCQXHbniGuOmcdoUDl/dvd+0e5aHM3nZFgyfZcvsC3HzpBMpOrOKYeBkd/SefskZJtPhF2bewiFvRDrA/OfyUAjxyb4OGjEzXPtX1tnOed1V+x/fETkzxwuGLAZBGb1sS46tx1FdvTuTx37x/lReeuK/mcjk8kmZjNsGtDd8P3t9pxy4PHGOyOcvn2vqVeCrBIBYkishW4BLgXGDDkoJQ6KSLmL20jpfMYjlnbSohERN6BVixs2VJvaJyHBcFP/y/c8+nSbdteCG/+bum2bBL+/TWg8qXbIz2w45rSbY9+E255Z3Pr6DkDznhOc8cAHP45ga+9jq+mP8ID6uzi5t/03cM/hT7J89P/wLEqBb47Bzs5a509dPHkZJI3fP5e/ua3LuTVl22u2B/gR4+f4v1ff4hP/86lXHfBYMlzX/v1UT50y6P8xcvP542Xn1Hy3J6TU7zh8/fyv6/fyduev63kuZ8fHOMPb364qbccIMcT4fcQknzlk09Yj+svKBLJH33zEZ48VZvUwwEfez56LT5fKTl+8FuP8Njxqbpr+a/3voCdG7pKtv3jTw7wqZ8c4PYPvJCz1tkTf//21n3cfWCEez90TflpPDhQKCj+8j/3cMXZa1cPkYhIHD0g6P1KqanyOzXnrlW2VXSUVErdCNwIcNlll3kdJ1uJdAIe/Ars3A3XfUJv+8/3w8iTlftOHtMkct3HYefL9c+fu0KTkJNIlIJffAr6z4Hf/U+qf+0OjB2Af70WptxMja2CmREA4pLk39/2bM5er8kh8tAxuB2+946LyK/dWdz9FwfHeO9NDzKVKlUA4zMZvRzrsRomk1kA/u8Pn+Sa8waKymM6leVvb9sLwPcePlFBJN9+UL+3o+OzFec8PDYDwI/efwV98ZCrt+ybeJrQv+RJvOhjpM95OaD/I73073/GW563jXdfeVaJuhufyfDyizfwv67fWXGur//6KH/zo71MJLP0dpS+/tHxJK9+xiY++NJKdZrM5Lnuk3fx97fv48Y32cMiRxNpPn/3U8V9nJhKZRmaSjM8nWJdZ8TVe12NePT4JGMzGV549oJ1OJk3WkokIhJEk8hXlVJmytyQsaxEZBA9mQ60AnHe6m1CT7zzsFR45GuQnoLn/HeIW3+0PWfAwTs0IThvCiYsG2XdTnvfZ/4e3PGXMPykbYUdugOGHoUb/hHilbZHBfyW1TN9am7vITUJQIgcgz1R+uNhvV3SAKwJA2YbMNCpfy6/yE1bxDJlkUU1zFrHHB6b5Sv3HOatlrr4pzsOMprI8OKdA9y+Z4hTkynWd+sLZb6g+M5D+s/8xESy4pxHxmYJB3ycPRCvsMtqYliL+PjmC4kPbLLXF+xljG77+7GQSOfoj4ftz8aBrX16FP3wdKqESGbSOSaTWbat7ah6HMDbn7+dv7t9H48em+SCTd3WZ3Gg+Dll8oWS/VOW9fjEiSnWnVNKJDPpHAWlKqy/1Yg7944gAle0EZG0MmtLgM8De5RSf+t46rvAm62f3wx8x7H9TVb21uXApBcfWUIoBffeCIMXw6Zn2ts710N2pjK+MXlMP/Y47gUuewv4w3DvZ+1tv/gUxAfgwnpTaB2IdEMgCtNz/FNIau8/TJZYyBHbyFp3//lSYoha+5QTScIiksm6RJJDBJ53Vh+f/Ml+JpNZjozN8oW7n+JVl27ig9eei1Lw/Uft93LvoTFOTaUIBXycnExVnPPo6Vk298bckwjYpN5Tav1GQ/7iRdwgly8wm8nXvECv69IkMTSVLtl+clKT3saeaM1lvOX5W+mOBvn72/cBOgby1XuOsK2/o/jaTpjP/PETlXbZB77xEO/+qjcJGODOfcNcuKmnQiEuJVqZtfU89HS5F4nIQ9a/64C/Bl4sIvuBF1u/A/wXcAg4APwz8O4Wrs1DIxy6E0b3wrPfWao8Oi3vv1whTB4F8UHnBntbR78mjIe/BrPjcOpROPgTfc5A9bvYCohA50DJ6z12fJJbHjzm7viiIskWSQKATEI/5kutKkM2yWyZIklrAim3vJyYzeSJBf186LrzmExm+fSdB/jrH+7B7xP+6NpzOGtdnJ2DXfznI7bQvuXB48TDAX7zgsHixdmJI+NJtvTG3L1Xg9OHQfzQtalkczTkL971GyTS+v3EI9XNiXWWQhueKiW54xP698Hu2kTSFQnyjiu28+Mnh3no6AT/YBHK+67eAUA2X+pMm8/8iZOlRJLNF7hr/yhDU5VEu9pweibDQ0cnuLKN1Ai0Nmvrbmob4FdX2V8B72nVejw0iV/dqDN7dr2ydHvnev04fRLW2sFrJo5qEvGX/Uld/t/gwX+D+78II3sh2AGXvbW5tXQOQmKo+OtX7z3Mtx44zg0XbcTva3CnbohEckSd2VYZo0hKicRkZM3V2oqGAuza0M0rLtnI5+96ilxB8YEXn81Al7Zqrr9okP/3w70cHZ9lbWeYHzx2imvPX8+W3hi3PHicdC5POKDXoJTi2Pgsz97WW/89lmPiCHRtrPguokF/zffVWZNI9LqHp8sUiWXDbeipH8t483O38i93HeJPb3mUPSenePNzt3JGnybGbKFMkTisLScePT7JbCZfQe6rEXcdGEUpeOE57UUkXmW7h0qcfhr2/gCe8bsQLLtQdFmKo9xqmjxWamsZDOyC7VfqoPtj34RL36TTgptB5/qS15vN5MnkCsVAdF1YRBL15Qj6HX/uGevYMmsrFtIX1ApF4sLaSmZyRUXzh79xDn6fMNgd4fdesL24z8su1J/f9x89ye17hkikc7ziko0MWjGTUw57a2I2y3Q6x6Y1te/6q2LiCKw5o2JzNOiv+b66ahBJNOSnMxJgpIxITkwk8QlFgqyFeDjAO194Jo+fmCIS9POeq84qfg/ZXFmMxCK5p8dmikoJ4J5DYwAkM6X7r0bcuXeYNbEgF23qWeqllMAjktWC2/4M7vzrxvsB/PpftE112dsqn4sP6McKIjkC3dXTYrn83Tp7SimtUJpF52CJtWXsmX1DCXufxAgcv7/yWItIOvxld7NZQySlisSolvJYwlQqW/JYDbOZfJFINvRE+fJbn8WX3vqsEkttc2+Mizf38L2HT/DtB08w0BXm8u19xVjDcUfA/YiVxdW0tTVxuCI+AlptVSoS/X7i4dpB7HWd4Qpb6cSkzqwqIecaeNNzzuCMvhjvueos+uNhm0iqWFsbe6IoBU867K17Do0DVNhyqw2FguJn+0Z4wY61jZX4IsMjktWCfbfCU3c13i+XgQe+DOe9DLo3Vj4fjkO4qzRGUsjD1Ano3lS5P8BZL4aB8+Gi11W9U26I+ICOaVgB/lRW35nuH3IE/H/2/+Arv1V5bC0iyVQnkkhQ/5cov3NPuLa2bNJ49vY+zh7orNjv+gsHefzEFHfsHWb3xdqeG7SI5OSEfcE+eloTyeZmiCSX1iRfhUhioUpFYu78a1lboO2tcmvrxESyoa1lv26An/7Pq3jPVWcBEPDri2CuirX1jDO0WjUB92y+wP1Pjxef1w746sTjJ6YYTWS4ss1sLfCIZPUgOQ75dOP9Zob1xffMq2rv07leE4fB9Cld+V7N2gLw+eAdd8INn2pmxY7XMwF+HScpKpJhhyIZ2avfY9nFiZTO2urwlRNJ9awtEbFiCaVBdTtGkqt5MZt1WFv1cP2FGxDRqb8vv1iTtbG2nAF3o0iaIpJi9lwVa6sKkTSKkQAMdIUZni5TJBNJNtTJ2KqHkKVIMg5rq1BQpLIFtvZ30NsRKsZJHjs+yUwmz87BLvIFVaFiVhPu3KsrJV6wwyMSD0sBpXTWVM4Fkczquz+idQK8ZVYTk1ZDglrWFuh6kLm0OIHSAD+QylVRJONWa5ZMghKYGIm/LNvKKJIqn0m1O3djAWXyhaIiKoe2thrnr6zvjvDcM/vYOdjFeYNWgWTQT29HiBOOGMnR8SR9HSHi4SZyYk4/rR+btbbqKZKuCMNT6SKBKqU4MZmaM5EYaytXsEkhbX2nsZCfnYNdPH5Sf2/G1jJ34as54P7TfSNcsLGbtZ0uMx4XER6RrAZkElDIVtx9V0XS6p1ULyBeQSTWXXA9IpkPylKOTVD20MiMrkXIpmwyq0EkMV8ZkdSIkYC+4JbHSKYdab+14iTJbN6VIgH4zBuewVff/uyS+pDB7khJUeLR8Vk2NR0fqV5DAjUIMm2C7fVjJOlcgamk3ndsJkMmV2BD99yqz421lXXUkZh1RQI+dm3oYt+pBNl8gXsOjbFjXZxNa/TnsFrjJJOzWR44crotbS3wiGR1wKgMN9aWIZJYPUViZVEZi8dcvGrFSOaLCkWSJ+ATMvkCT4/N6uCy6abjLJTMpiCn7/CjFdZW9awtqF5v4SSSWplbzmB7I3RFgqwpKyjb0BOtiJE0H2g/Ar6AnV3nQK3036BfCFdpJGmwrsukAOu1mTXOV5E4bSpDJNGQn50busjkC+w9Nc19T4/z7O29RENW7Kps/T95coi79o/MaR3LCXcdGKGg8IjEwxIiaRFJrnafqIp9GymSQhZmdVomk0f1/uF47WPmg3AnBGPFWpJUNs85Vs+s/UPTMHbQ3tdJJGk78yci5dZW9ToSsO7cyyvb07licV6tgPtsOkc0OPfSrA3dEU5YMZJcvsDx00k2N536e1gTehUb0aT/FhyW0nQqSzwcqFs5XyxKtALux4s1JHMlkkpFYog7EvSzy2ry+I37jjKTyXP59r5iNl25ovrErfv44DcfKXlPKxG3PzFETxum/Rp4RLIaMBdFUo9IuozVZKUATx5rna0FVnW7XUuSyhbYtaELESsFeNxJJI5itqTdGr2ESJRyVLZXkkI1a2sqlWWjdVGvZm0ppZhtwtqqhsGeKNOpHIl0jpOTKXIFNTdFUsXWAoha8Zu0I8idSOUa9q8yRGJSgE/Mm0isGInT2rI+72jQz7b+ONGgn5vv05bps7f12YWiZUQymcxyYjLFr6zMrpWITK7Aj/cM8+LzBgi4SLdeCrTnqjwsLAw5uFEks+O6t1WwzkWivE3KxNGaF68FgyMuk8rmWRMLsXlNjH3D0zB+yN4v7YiRWPERgJA4Lv65FEUrrIYicVpbhYIikc4Vaz2qWVvpXAGlIBaeB5GYzK2J5NxSf0G3R6mSsQUQrZLaPJ3K1c3YAqe1pW9ETk4mCQd8rInNrYFiwKqByDisrZTD2vL7hHMHO0lm85y1Ls7aznBRkaTKCd76LkwH5eWAA8PTPHZ8svGOFn5xcJTpdI5rz1/fwlXNDx6RrAYYInGlSCYaV56bmMXUCX13P3m0dfER52tOn6RQUKRzBcJBP2cPxG1rq8Pyjp3WloNIwk5FknFUxFchkmiZIpnJ5FCKYsDXBJ2dMPvHagy9cgNnUeLRuRQjZpM6fbsWkYRMsaW9fjdEEg8H6Aj5GbYaN56YSLGxJ9pcI0kHRISgX6oG2w1h7BzU9tbl23tL1p4sI3iTLPD9R08ui0C8Uoq3fek+rv/U3fz2P9/DLw6MNqyN+dHjp+gI+asOF2sXeESyGmCsrVzaDpDXQnK8fqAdIG6C36d0nUYm0Vpry7zm9BBpxwXn7IFODo3MoMYPweBFer8SItHW1pSKEqIWkVQPtpfftQNFa6uaIjEXZzfpv7VQLEqcTHF0PFlsseIaE1bmWgNry3nBnUpl61a1G6zrihSD7Scm515DYhDw+apaW8bCMlMSzeCmajGShEXwV52zlulUjp88OUy749DoDIfHZrn63HXsH07w2/9yL6/93D2MJqrf5OULilsfH+JF5w3UnMzZDvCIZDXABNBRugq97r6nGyuSQAhi/TpmYS5ei6FIsjOkZrTKiAR9nD3Qib+Q1jEaQySZSkUyonoI4bj4u1AkzmC7qf5eEwsSC/mrBtuLHv88YiQDnWF8oq2tI+OzbOiJNOeJTxzWjzW6BxQvxo6eVYl0rmafLSfWdoYdiiTZHMFVgVYk1bO2AH5j1wC//ewtXHWOnllTrZmm+R5esms96zrD3LIM7K079+oMs4/csIu7/ugq/mL3Lh45PsHrbrynosMywH1PjzM2k+HaXe1ra4FHJKsDs45AZCN7yw2RgBWzOFl9DkkrYMVlspP6YhEJ+tkxEGeTjCAoWHsuBCJVFcko3QSV4+KfdUwirEUkJYrE9KMK0B0NVlUkM8bamgeRBPw+1nVGODGZmmPqr0UktRRJcG7WFuiA+/B0ikyuwPB0et6KJBTwVc3aMmvsj4f5P6+4gA6rGNMQjFNNme+hJxZi98UbuHPvMKfrTLBsB9y5d5iz1sXZ3BsjEvTzxuds5YtveRYnJpK89sZ7KkYJ/PDxU4QCvrZN+zXwiGQ1IOkgkkbV7bPjLonEyqIqVrW3Otiu78jykzpzKxL0cebaONt9VsC/90ydJlwWI1H+MNMqSkA5LjDOosUq1pYp3DPe9VSxjUiQrkiwatbWQlhbAIM9uijx6Pgsm9fMIWPLH7KtxzKUxxmU0kkE9araDQa6dL+toakUStUfaOUGAV8pkTiztqquvYq1ZWJVXdEAr7hkE9m8Khka1m6YzeS499A4V5WRwuXb+/jyW5/FyHSa13zul8Wu1kopfvTYKa7YsbZIqO0Kj0hWA0oUSZ07NqW0ImkUIwGdAjx9ShNJIKKHWLUSliIpTGniiAT8RIJ+Lumw3lvvNgjFK7K28qEuMgQJOBVJxqlIKok1EvKjlJ0m62y1XkuRJBdAkYBOqT04kmA0kZlbxlb3Zt3brAqKmU/WxTiZzZMvuBtfu64zzGwmz/5hTdSDLhs21kIwIORKrC39WdeyBiNVbDlD6F2RIOcNdnLOQGdb21u/ODBGJl/gynMqR0xftrWXr7z92Uyncuz+p5/ziwOjPHp8khOTqbbO1jLwiGQ1wK0iMa1U3FpbiWHd46p7U+kUxVagU7evV9NGkVjZPeFRpuiEWC+ZQJyTI8NFK4rUJLlQJxkCBApORWLFSHyB6um/ZX68OV9nJEhXNFA/a2u+RNIdKY61XcgaEqhUJG4aNhqYkbsPHdF24XytraDPVzKz3aypVoW93yeEAr4yRaK/l+5oEBHh6vPW8cCR021bnHjnvmFiIT+Xba3+/+vizT185z3PY208zBu/8Cv+93cex+8TrjmvknjaDR6RtAtyGfji9XDknoU/9+xpuwljPUXiphjRoHM9oOD4A63P2AJtW4Xi+KxaEkMk23ynOFRYx+5/+jkPDOU4fGKI/zL2RmqSbLCLtArhK4mRWEQSXVMzawtg1rpoJRwX3K5IfUUyn2A7lI6ubVqRTByu26Y/Vkz/LSVIN00hzaTEB49aRFJnxK4bBP2+EkWSyuaJBv11U4qjQX9Zxllpn7DejhBK6XTtdoNSijv3jvC8s/qLEzCr4Yy+Dv7j3c/lyrPX8vDRCZ6zvY+eWPvMZq+FlhGJiHxBRIZF5DHHtq875rc/LSIPWdu3ikjS8dxnW7WutsXsGDx9Fxz55cKeN5+D9KRdRFhPkbjp/GtQLEqsM4dkodG5Hv+MbpNi5oYM5E7wlFqPUoq1fX3ESdp9sVKTpANakfirKZLomurBdjMlMWPfuftEX4i7otVjJDMLFCNxzvhoSpGkE/pvqI4iKc98mk41bthoMGApkoePTrAmFpw3YQbK60gy+eJ3Wgvl2XRGkZgYjyFE53TFdsHBkQTHTiddBc07I0FufNNlfOwV5/Oh685bhNXNH62M4HwR+Efgy2aDUuq15mcR+QTgLO88qJS6uIXraW9YzQWdRXQLAqMyOtfD8OP1s7aaUiSD9s+trmp3vGYwoWsFIkE/ZFOEZ05w/Qt/h1dc83zyN6/nyNh+uwVIcoJUdIAMCp+TMEyMJNJTXZFUsbZMP6quaJBEOkehoPA5ptQtlLVlFElHyN9c5bhJeqhRjAiVMZJmrK21liKZSuWKvbDmg6DfR7ZQmv5bK9BuUF7fM5XK0hkOFKcFGkJJpHLQPe8lLihM2m+1+Eg1+H3C7zx7DkPglggtUyRKqZ8BVRvgiNavrwFuatXrLzsYpZCaqr9fszDxEXPhr9dK3k3nXwMnkSyWIokPEEoaReKHicMIiuC6HQD4Il10StK2P1KTJP1xMgSQQlnWlj8EoVjN9F8ojSWYgHR3NIhSpd2AQZOOT2p7/G5hYg+be2PNVY6fNqm/tS8+Qb/g94nD2tLvwU3WVlckUHxvg/O0tcxanDPbk9k8kQYkHK6IkeToitpkaxTJdBsqkjv2DnP2QHze2W7tiqWKkbwAGFJK7Xds2yYiD4rIT0XkBbUOFJF3iMh9InLfyMgKah/dKkVi7CrT1qSeteWm869BRz+I9R9/MWIkAJ3riSRHAKVtENP1t3c7ABLpJI5FJEpBapJZX5wMQSSfsav6s7O6m7A/VMPaKiWSKUethSneK7e3zFCrubYNMejrCBHy++YWaIe66lBEiDlqZBJpO4mgEUSEAavn1sZ5ZmyBpUgc1lbapSIpryNxEkmnU5G0EWbSOX791OliceVKxFIRyespVSMngS1KqUuADwD/LiJV9bNS6kal1GVKqcvWrm3vIp2mUFQkLbS2YOGC7T6/nqUOrS9GNOgcJFBI0cWsvuiMlxIJoU4ikiWTyei+U4UsM9JBWll33OYzzszoVGF/qIG1pS9IiXS2GEcwF67ygHsym5t33ADA5xNe96zN/OaFg413dmLisE7Djte/WEUcF+NmrC2wuwDPN2NUVo/HAAAgAElEQVQL5mhtlcdIUtmSqnxDiOVqcanxxMkpMvlCsd3LSsSiE4mIBIBXAl8325RSaaXUmPXz/cBB4OzFXtuSomUxkjJrq26w/TQEOyDgcpRn1yAg0Fk5RKklsMhwrUxoa2v8kCY9Y8WF9YwSlZoufo4JiZEV66417ySSmB7/W6P7L5RaW8b+6baIpLxNykx6fi3knfjo7vPZbc1yd40jv9TV/Q0UkbMhpcl66nCZIGBSgAcXhEjKrK1MviERl3ccmCpTJHawvfS7uXv/KAeGyyZnLiIOjejXPnNti+b1tAGWQpFcAzyplDpmNojIWhHtk4jIdmAHcKjG8SsTrVIk5dZWo2C7GzVi0DmoJ/EFFik90SLDzTKs/fqxg7YagSKRSHba0bCxA/wWMZo2+pkZCHU0tLacsQTb2qquSNzOa28Jxg/B8fth1ysa7uoc2pVI5Yg7gtWNYFKAF8rayhWcMZJCw6aEkTJrazqVK8k4M2Rfrkg+8I2H+Lvb9s17zXPFoZEZQgFfsennSkTL/vJF5CbgSqBfRI4BH1ZKfR54HZVB9iuAj4pIDsgD71JKrdxJNdXQSkXiC9gEUW8mSXIcYk0QyZV/AjOLGKfqP5u0L8Zng3+P3JqE0X2w1RFOsyY0StpWJJOqAxUI678qQ6LZWa28/MG6dSTOrC1DJN1WJlV5jCSZzS2YImkaj31LP57/qoa7Rhx39c735QZFRbIAwfaA31fStDHl0tpKZR2V7cksXVF7/UZZOdN/CwXFaCLNwZGlUyQHR2bY2hdzTdjLES0jEqXU62ts/90q274FfKtVa1kWMIokvcBZW7Pjui7E2FULqUjWnz+/tTWL+Fo+c+4X2fbEP7H7nk+DKlRVJL7sTJFIThdiiD+kiaQYI0noflT+UFWrz5n+q5Qqydoynnw1ReKmsK8lePRbsPlyV7EqZ5zBbcNGg1desolo0D/vzr9A5TySjMsYiUWCeWsWiVOR+H1CR8hfEmyfSGYpKN2+PV9QS3IxPzSa4Ox1nYv+uouJ9u4EtppgFEl2VquGhbKLzHwRY+80Sv9dt3NhXrdFOOFbz02h97L79/4GHvgyXPRa+8mwzs/wZ6cdRBIlFghDBtvGysxaMZJw1c8j6PcR8AnJbJ50rkCuoIoX3Hg4gE8qh1slM3nWxl3GlhYSQ4/DyB647uOudo+F/Jya0u85kc41RX7ruyO85Xnb5rTMcgTLmzZmXcRIymw5oCRGAtreciqS8Rl9o5DJFTh+OsmWPjsb7tBIgvsOn2bfqWn2Dyc4oy/G7os3cOmWNfPOvjPI5gscGZtt+zbw84VHJO2CnGMWQXoKAgvUBNG0RzHE1KiyvRlFsgRIZQv6znXtOfCSj5U+GdLWVsChSMbzUeJBEyNxBts7agbbQV+0ZjP5ooXVaV1wTVFi9fTfJbC2Hv0miA927na1eyRUam0tVfuNyqaN+cYxkqDdldlu2Fh6CYuHAyV1JGMJ+/s9OJIoEkkqm+c3P3k3yWyecMDHtv4O7jk0xpd/eZiNPVHed80OXnNZbYWXyuYRoW67E4Bjp5PkCortKzjQDh6RtA+cF7TU5MJ1002Oa/vH38Daaqbz7xIiVe+CY1lbwdwMpPSd6mguwragZcWYzzg7Y8VIqgfbwe7rNO1oIW9Qrd/WbCZHbLGtLaV0fGTbCxum/RqUW1tN9/NaIAQcTRvzBUUmV3BlbYHuymw+/0pFEiyxtsZmSonkqnP15/TY8UmS2Tx/81sX8spLN+H3CYl0jlsfP8VX7jnMH33zEY6Oz/KBF59doU6+/8hJPnTLo7x45wAff/VFdddsMra2r+2ou99yh0ck7QKnIrEyjhYEs+Ow8Rn6ogm1g+3pKVD59lck1rz2qrCIJJRPQBIIxpjO+vBFaymSkH7Phbyui3HAzCSpVmvRHQ1WpP/OZvLzmtfuGk98FxJDsPPlunZk4jC88I9cHx5zKJKpJmMkC4lQwG7aaDKxGvfa0s8nM/ni51/eJ6wzXGptGSIJ+ISDI/ZkzAeO6Jqpq85dZ7dYCQd45aWbuOGiDfzpLY/xqZ8c4MREio/u3oXfJyQzef7i+0/wHw8cxydw71NjDd/nIes1t/d7ROJhMeC0nBaqTYpSdozE5wNfsLYiaaYYcQmRyuaJ1GpDYllbofwspPIQ6WY2k8dvVWSTS2siLeR0jMRU5uezFUQSseotnC3kDbqigRJFopQimW3S2kpNwYHbXGVaFfHw1+CWd+qff/BBnQ7tD8G517s+hVORJNJZV1XtrUDAZwfby8fs1oKz44CxtrrLFUk4wMi0/Tc+Zs1C37WxuyRz68EjE2zpjdFfJa4V8Pv461ddwGBPhL+/fT/feqBYqYBP4L1X78Avwt/dvs8qiqz9GR4aTdDbEVoWHXznA49I2gUlimSBUoAzM9q6Md18A+HaiqSZzr9LiHQ2X/s/pc9H2hcjUpiBVEoTyek8gZCxttL2dMRQ3J5fn89AsDQTKWbVLCRqKBIzMwR03EYpu2uwKzz6Dfj+H8CW5+hanEbY8z349rt1uvNv/CU88W147D/gotdDtMf1y0aCftK5AulcnlS2UIz9LDZ0HYnSJJwxiqRxjAQsInFMR3QiHgnY82iA8ZkMPbEg5w508uMndZ82pRQPHDldt9JcRHj/NWdz0aYe9pyyb+yed2Y/F23u4Y4ndfPQJ09O86xttf/PHByZWfFqBDwiaR+UKJIFIhJT1W7iHv7QslckyWye9XUskIy/g0guWVQkyUyOQMiqe8il7XntwZj9mdeoJdGKxGps6LjglsdI7DG7TSiSmVH9mJpsTCSH7oRvvhU2XAKvv0lbeBsuhms+4v71LJg1mrt2Nw0bW4GQpSqzeVUxr70WnGnZxWB7FUVSHmzv7Qhx5roOvn5fhonZDLOZPENTaS7d0vhv/apz1xXjKk7stDogP3Fisi6RHBqZ4UXnrqBWTjXgEUm7IJfSF7fs7MIRSbnKCIRrZ2010/l3CVHM2qqBbKCDWGoWlUpBfB2z2TzBkCPYbmaRhDp0HYrZXoZoMMDpmWTJOFeDrrIYyZxayJvvJu2iUO62P9PNGH/n5mIcaK4w9tCwRSRLaW2BTo81RYZumjaCtjenkllEIF6mAjut9F+lFCLC2Eya/o5wsT3JwZEZTk4mAbhki3slV451nWH6OkI8cbK2DT2VyjKaSK/4jC3wJiS2D3JpiPXpVM4FUyRl5FCjkrtk3zZXJHWztoBcoIM4SVRygkKoC6UgGHbESJxEYhIQarRJcQbb42XWVjpXKN5J20TSxH3ZrBWobVSAWijA6H7Y8ZIFIXnz2Q1PGSJZOmsLIJdX7mMkTmsrlaMzHCiZCQNakShlfydFRWJdzA+NJHjwyAThgI/zBuc+V0VEOG+wqy6RrJZAO3hE0j7IpXT31ki3eyI5dCfc/6XazyfLFIk/vOytrYZEEozTIUk9ZjekLxShiGVtlSuSIpFU6wDsI2lZWx0hf0lFdHkr+TlZW+a7yTRQJNMntUrt215/P5cwF+ORaR2TW7oYif48M/lCkUhcx0isrK1yWwscw60se2t8JkNfPMSmNVGCfp259cCR01y4qbtIZnPFzg1d7BtKlBRWOmGn/nqKxMNiIZe2icRtm5T7vwQ/q1PRPFsWI2kUbA91atXSxtDpv7X/bPPBOJ0kkfQkmYC2gcJORVKMkXTYRZpVyDUWCpDM5qtmNnWVdQCe07z2oiJpQCTFVvlnuj93HcTaxNoqKpJCwf78XFpbJmurWraUs5V8vqAYn83Q1xEi4Pexta+DPSenePz4lKv4SCPsHOwikysUlUc5Do3M4PdJ87NlliE8ImkX5FL6Qt+MIsmlIZes/Xy5ymgUbG9zNWIK1yJ1qolVqJMBOY2oAmmLSEIR6z9ySdZWfWsrYqXJOlvIG9gzSfRd79xiJNZ3k56uv58Z3tV3lvtz10G0TaytgEUk2Zwj2O7S2kpl89ZQq8q1dzrmtk/MZlAK+qwU3zPXxvn5gVEy+cK84iMGxYD7Sf3/NZsv8G/3HGbUSjk+NJpgS2+smFiwkrHy3+FygVEk4a4miCQF2VTt52fH9fmMyqgbbG+y8+8SIJ1rfMFRoQ7WiCaLlF9bCpGSGImlSMw8EqhqbcVCfjL5AhOzlR1yzZ2wsbZmitbWHGIkmUZEckD/XXQ1OZ+kBiJFRaL/bpYqa8tYW9mCbW01lbWVzFVVJM657eNWMWJvh75hOHNdBzlrmNYlC6BItvd3EAr42HNSf4c3/eoI//vbj/HurzxALq+VymqIj4BHJO2DliiSst5ZdVqCLAdFYrJ7ahYkAipsB1BnfZpIYuGgLsYsCbbH6wfbg/YFt8LaKpt7kWxWkeTSuk0LNFYk44dgzTZdULoAqLS2ljbYns3PxdoqaGurWozEMdxq1Oqz1Re3iMSKVWzsiRbHBs8HAb+PcwY6eeLEFDPpHJ/88X7Wd0X41dPjfPzWfTw1OrPiW6MYeETSLijGSHqaUySFXO1MrPImjI3Sf9u8GDHlIigrjvRYQyTRkF9/tvmMfQE3M9uh7nCr4al0xcXW9uFNsL1JIpl1jNppFCMZOwB9CxMfASdBpgkFfA2bDrYKQYe1VQy2h+pfjsLWDUTSSv8tr2oHm0imHYqkr8O2tgAuXgBby2Cnlbn1z3cdYjSR4TNvuJTXPXMzn/3pQdK5wqoItINHJO2DEkXiMthuSCFbQ5WY9igGNWaUA8uk829jIvFFbCJJiKVIQgEdWDeKRPz6s65jbZkLrp55UVk9DXYrc7fpq0XMOno01cvaKuTh9NMtIZKxRLrifS0mnNZWKpvHJxBqkEUlIkSDet7ITCZfI9huE8mY1ULeWFtnrYvTGQ7wwh0LVyC4c0MX4zMZPnPnQV56/nou2bKGj9ywi3PX67/D1WJteQWJ7QJn1lZmGvI58Df4ekxblVwKqJITPzuubRGDWsH2QkE3ilwGxYhQv7mfL2J/DlPEgIxWCib1OTOrA+0irhQJUDGzoyPkxye2tTWbyeH3ScMLYRFJpyKpY21NHtVrW6CMLbDfV0FVvq/FhK1ItLUVCfpdzQCJhvzF+E61YHuHI9ieLyhEYI011bIjHOCXH7qajgVs929qUXIFxR++5BxA3+h89g3P4As/f4qLNi+c+mlntEyRiMgXRGRYRB5zbPuIiBwXkYesf9c5nvsTETkgIntF5CWtWlfbwqlIwF0KcLOKpFb6b3pSV3m3uSIxd/41u/8CgZhNJNNK3w1qayuk33smoYkEXBNJeYxERHQrDoe1FQu5uxDqAyxFEukuJRKlSq2usQP6cYEytqBUzS1V6i8403+1tdUoPmIQDfqLGWfVFEnQ7yMS9JFIa0XSEw0WM8RAk+dCDa0COG+wk5Dfx2ufublonQFs7e/go7vPb1gbs1LQSmvri8C1Vbb/nVLqYuvffwGIyE70LPdd1jGfFpHV8Q0YFBWJdSF0EycpUSRlyOf0OaLl1lYVRbJMihHTLrJ7gtHu4s8TBV2IWKJIsrM6PgK1ra3MDGft+TQBKhs2GnRGgrYiSTfZ+dfESHq2lFpb+2+DvzkTTh/Wv48d0o8LaG0F/b6irbRUgXaAQFlBotsLbiToY6ioSKoTYTwcLMZI+lo8tbIzEuS/3vd8Pvyy9p4s2mq0jEiUUj8DxhvuqLEb+JpSKq2Uego4ADyrVWtrS5QrknIiGTuoO8CWHFNHkZiZJhWKpAqRmJqGdg+25xrHSIKWIsn6Y8xavfsiAacimWmsSA78mM0P/R2XiFYE1e7cOyMBpgyRZPPNpf4aa6vnjFJFMrJH/x3s+5H+ffygzi6LD7g/twsYIl5KayvkaJGScjFm1yAS9DM0ZRFJDSI0/bZGrfYorcZZ6zqXLGmhXbAUwfb/LiKPWNaXuQXeCBx17HPM2rY6UCjoi5mJkUCltXXPZ+CW/1a6rZ4iSVYhh1rpv8tEkbiJkYQ69OeXCnQym9GWic8njhiJk0jM1Miyz2RWd+btsepRqiuSAIm0qWzPubZm9PnHNUHEekutrIRuTc6B2/Tj2AE93XIBrRiwbbultLaMIjHpv81YW+bvoLYiCZBIZRmfydAfX9lzQNoFi00knwHOBC4GTgKfsLZX+5+iqmxDRN4hIveJyH0jIyOtWeViw9hN9RRJ4lRpzUihYB9XTZGYNuUxF+m/y6bzr6VI6tz9hWIWkfg7LaVg7WveewmRWBei8rjRjI5hdItOFa7Wj6rE2mp2XvusFbsKd5UqEkMkT/1Mf6djBxfU1jIwF+2ltLacdSSNOjo74VQudYkknWMskV4UReJhkYlEKTWklMorpQrAP2PbV8eAzY5dNwEnapzjRqXUZUqpy9auXSF9/o2icCqSCiIZ1jUjBdP6PF15POjnH7oJvvEm8AWgb4f9nD9sj5Z1otjcsb0ViZvmfqaOJOmP6zvdciIpiZHUsLYsRdKNUSTVra0SImnGJpod00oxFNd1LeY7TQxZhZMpOHiHHqO7gIF2AzOAa0mJxGfPI0lm88WK+0Zwfve1rK14RE+wnEhmizUkHlqLRSUSERl0/PoKwGR0fRd4nYiERWQbsAP41WKubUlh7ogDYX2XClWIRE93o2AFhp3k4VQkX3s9fPtdOpD79tthzRn2c4EaF06jSCLtnaroxtoiGCWHj1npYDaTs5VCefov1O7+a2VV9RhFUsPamnZ0/21qXrvJpgtbWT4m4D4zAtuvhEAUfv3POpNuAVN/Dczs8yUlkoA2IXJ5XUcSbTCv3cAoF59AR424VGc4wNHxpNVny1Mki4GW/SWJyE3AlUC/iBwDPgxcKSIXo22rp4F3AiilHheRbwBPADngPUqpfLXzrkg4FUm4C5BSIlHKtj3ymUqLyhyfTcK+H8Jlb4XrPlHZVsPEBHJpCEbt7YlhTSKN6laWGG4KEhFhhhgzvriOkZiLTbX0X58fkEpinTGKpB6RaGtLKTU3a6t3uz2kKj2ts/USQ3r07rYrYL8VcG+FtdUGMZKSFilNpv+CtrXKZ5EYxCOBonr1FMnioGVXDqXU66ts/nyd/T8GfKxV62lr5BwxEp+vsnFjJmG3P8/XUSTmmIHzq/dmKqa7ll04x/a3xEJZaKSzeUTsVhm1cHPwBnLxi0hm8rZSqJb+a4oSa1hbfT5NJNUaG8bDAXIFRSpbKLXQ3GB23La2QH+/+axWQvF1sO48B5G0wNpqg6wtY21l8qqpz8/sV62GxMBJ/F6MZHHgtUhpBzgVCVS2STFqBGzScSqSciKJ2LUUJQg4FIkTI3th7TnNr3uRkcoVCAd8DQvKbo69ngfDl5UqhUBIf065lH0Bh+ptY6w6jzW+2Zr9qIqNG9PZ5hRJPqsLQGN9DkWS0LYWaCLZ8WL9c6SnJQkQbREjcVhbzdWRGEVSe+3xsE0ynrW1OGhvL2O1oKhInETiUCROIjF3z05FYrK5klbtSK1YR7V01+SEtlT6z57b2hcRppVGI0SCPq0UnPUJ/rAdCwo5Bg35g6Wfh1JFa6tHEnVqFcxwqxzJZupInBlyRSKZgoS1zvgArNkK/efUviGYJ+wYyRKm//qcWVtzsLbqrN2pIPs8RbIo8IikHVBUJNaFvoJIhuyfi9aWU5FYx5tjojWIxATbnceO7tOPa89tft2LjFQ2Xzf11yAc9JPK5kuD7YGwTRghRyO9cmsrkyhmxHWTqHmxNXfzpu9T051/o2sqrS2AjnX68dVfXPD6EYP2SP/V7y2ZzZPNK9eKJGp1CK5rbVmWnQj0xDwiWQx4RNIOqFAkXTDhqM9sqEgMkRhFUuNOtqhIHEQy8qR+XNv+iiSVK7jy0iNBP5NJYzlZf+J+xwUlWE4kDmvL9MEKxenMztS82Jr4gun75JpITKp1rM/O2konoGB9d3GLSAZa13KjHawtESHgk2IKdfPB9nrWln6uNxbCXyMg72Fh4cVI2gFNKRJDJPViJA0UifPCObJXE0zPGdWPaSOksvmGgXbQg6/S2Xxp642AY5CRU5EEyhSJVYxI35nEVYId/dXnbRulYtp1RN1aW4aoTEEi6Kwt8x0bImkhbEWydNYW6MwtQyTN1pG4sba8QPviwVMk7YCqwfZG1lYVRZJ0qUjKra3+HVYqbHsj5TIoGw35mU7lyOaVnbUVcFxUSmIkZURiZWzRdxa+kw/z8RscbfgdMHfzQ80qklmHIilaW9M62B7uLk3LbhF2X7yBjrB/SbO2QLdJmUrqv+dmK9trVbWDrUi8QPviwVMk7YCitWX94cfX6cweMxa2xNqql7U1oVNbAzX+AwWqWVvtk7GllCpOtauGdLZQvxjRQiTg5/SsPk9JsN2gJGurLNhuFIOVdivGLiyDuSMeajpGYp0/2mulewdtRRJfnE4NW/s7ePsLti/Ka9VDyKFImg+21yZBQ/JeDcniwSOSdkC5IjE2k4mTJIYgZGX4lMdIwt2lMZJ6mT7lvaUyszBxRGcItQF+9PgQl//Vjzldg0zcpolGgj7H+FtTkOi4qATrKBLTo8y0ljFZVmUw9snwlCESt1lb4/p7DsV0NDgc1zGSxPCCd/ltdwT9Pqas7gDRBmN2DQyRdMc8RdJO8IikHeAsSAQHkVhzKRLD0GO1Iiu3tqLdpTGSem1OyoPtY/sB1TaB9oMjCTK5AiOJ6nPl3WZtOcnGbpHitLbqBdtH9bbuTfr3GkTi9wmxkJ+hqTRnynE2Hrq54br0+ce1rVVcS6fO2koMLUp8pJ0Q8NvBdrdZW4ZA+uvMGYlHAoT8PtZ3R2ru42Fh4RFJO6BCkWzRj6cP64Z+M8P2ha082B7pcSiSyfqKpFiQaJ1jxEr9bRNFMjKt35O5uJQjlXNXAe2coFjStNGghEiClcH2WL/dwLKGtQXaQhmaSvE/At9iw91/bDdfrAdT1V5cbKdlbY3Yqb+rBCGnInFJJDsHu7jp9y7neWf219wnHPBz87uewxsvb/8EkpUCj0jaAYYUjGKIr9OkMnFY3xEXclWIxCiSHluRJCdq15CAo0mh9XojT4L4W9LPaS4Ysywt0wyxHCm3MRLHPu4USVmMJNZnE0kNRQI66ymXy3KF7xFEFXTQvBHKxx+H49pOS0+uSkWSSFsxEpcxJhHhOWf21eyzZXDR5p4lz0pbTXBNJCLyfBF5i/XzWqtLr4eFQC6lL2imP5aIViUTh7UaAQeRlBUkRrrnoEisY0f3Qu+20rv1JcSopUjMxQXgwSOni8Si039dWFuBKtaW8z0GHJlR1aytjj6bkOsSSYBnyD66xOqDlqytXuzzj5USSSiuJyHCqoyRKGvqUFODwTy0HVwRiYh8GPgg8CfWpiDwlVYtatXBzGt3oucMbW2Z1N/uzfa+YJFPWAeOTUPHhsH2sjqSkX1tY2sBjCZKra1UNs9rPvdL/v3eI8Xf3QXbHdZWsCzYHuwobWhZYW2NamsrYH22hhwSI3D66ZLX6YwEucr/kL2hvPV/NVSztpx9tlYRAn77e/CIZHnDrSJ5BXAD6L7aSqkTQGerFrXqYOa1O9GzRWdUmdRfQyTOGEkwogkom9L+fGqqfrDdmf6bz+o74TZJ/QXb2kpYRDKaSJPNK8ZmMuQLymqlMVdry3rvobICwwpryxEMj66xieT7H4AvvLRkKFhnOMBVvoeYUda5GxFJIa8VjjPYHnakIq8yIgn5bXvKbUGih/aEWyLJKKUU1vhbEelosL+HZlBNkaw5QyuM0f369wprK6WPCUb1z+kpQDVQJI5g+/ghHXtpEyLJ5QvF2g9jZY0XYya54iwSN3euVbO2DImGyv50ndZWLqNjFR1WIDe6xra2jv4Kpk/AUz8tHrrJN8a5vqPcIc/UGxoRSWoSUGUxki7751VmbQV8niJZKXBLJN8Qkc8BPSLye8Dt6FG5HhYCtRQJwLFfa0/fXHyciiQQthRJsnHDRtCWji+gFUmxx1Z7EMn4TKbol09bMZKxhE0sroZaWXAqkmh5sD1YjUisz7TYvsRSDJEeTeZTJyBxSm975BvFQy9O3QvAjwNX6g11Mrz0+R1V7QbO4siOFTI62iWCVrsbv0+Kg648LE+4+vaUUh8Hvgl8CzgH+DOl1KdaubBVhVoxEoDj92vLo1gDUkWRFLL2RbBR63F/SL/e8B5A2iZG4qwdMTGSotWVzpHKuRiza6E02F4WI6mqSGoQSbRHK5ITD+rf1+2CPd8rdhw4d/oejhTWcjBynn7eqUjGDsJDN5W+lrOq3cBYW9Feu2B0lcBYW54aWf5wG2zfBtyllPqfSqk/BO4Wka2tXNiqQlVFYhFJekoTSflYWKNITG8mE5RvNHfdXDiHn9AZW+UxgyWCUR9gx0jGZ+zgezOKxNSRhAI+u/urUSQVMZKgXVdj+myVW1vHH9Bp0i/+c108uPcHcOReNk/8ijsKF6NCnVSMR37gS/Dtd9nnBkfn3zWOxVqhxlVma4FtbbktRvTQvnCrJ28GnNVWeWtbTYjIF0RkWEQec2z7GxF5UkQeEZFbRKTH2r5VRJIi8pD177PNvpFljWqKJOYYxRofcIyFdWRtBSJ2Kuu0Zb00UiRm3vvwHljXulblzcJkbPV1hJhOa9XlDL4nrZYnrtJ/LdVS0v+qqEjipTs7FYlpjxIzRNKjg+0nHtCf1ZlXQ9dG+On/hS/vJhkZ4LO5G4iGQpXjkY2N5Wy4WU2RmNY3qyzQDra15bY9iof2hdtvMKCUKt5aWT83amTzReDasm23AecrpS4E9mGnEwMcVEpdbP17l8t1rQzk05WKxNSSgH2RCYRL60gCEZ25BfYFq16MBLRFlp7W1su68xZm/QsAQyRb+ztsRZKwra10ziiSxn+yxiqJOe90jTUYrJK1pfI6o6o8hhFdo6dPHrsPNl6iY0wXvFp3TF57Dr9+0dc4SZ+Ow0S6S+tITJDeSSSGqJyxEGNtrUYi8XnW1kqBWyIZEZEbzC8iskH+GvIAACAASURBVBsYrXeAUupnwHjZtluVUqba7B5gUxNrXbkw6qIcxt4ytoez5sHYYUVFclI/NlQkIRh6TF8824pIMoQCuj9SeYxkKpUlldWCuJmsrUhVRVIeI7HiEvmsZW2JndhgqtvTU7DhEv3zc98LV38Yfvc/ifTo7yUW8uueZ05FYkjFKEXQ9SKBaOkaVrG1ZQLsHpEsf7glkncBHxKRIyJyFF2c+M55vvZbgR84ft8mIg+KyE9F5AW1DhKRd4jIfSJy38jIyDyX0CbIVVEkUKlInDZMuSKZHgLx2VZJLfjDdkpxm1lba+NhuiIBO2vLEWw31lYzBYnVra0qwXbQn+vMqCYPM5vFGW/acKl+7OiDF3wAwp3FFhxakfSUEYmlSAzBgz5/x9rSEbqh1atIAlaw3YuRLH+46n2tlDoIXC4icUCUUi6aCtWGiPwpkAO+am06CWxRSo2JyDOAb4vILqXUVJW13AjcCHDZZZep+ayjbVBLkayppkicWVtliiTcVVq1XQ2BEKD0HIze9uixBVqR9MdDdEaCjjoSbXcpZdeUNJP+Gws6/rz9Ia3WOteX7uys9jd9tgyMIvGHqpKumXvREQpAoVvX5hiYVOASa2ukcuZIfEAH8nuXfj7IYqOoSLxixGUPV0QiImHgVcBWICDWHZVS6qPNvqCIvBm4HrjaKnJEKZUG0tbP94vIQeBs4L5mz78sUUuR9Fvt3Y0yqatITjWOj4AdK+jfUXsA1hJgdDrN+u4I8XCAVLZANl9gPJEhFvIzm8kX04ObSf8tuUCJwH/7hR1INyhaWxlNJB2O5w2RrL+g6mdlZpJo5VNLkTitrWEdrHeiaxB+/z7o2drwfa00BL303xUDt9bWd4DdaBUx4/jXFETkWrQtdoNSataxfa2I+K2ftwM7gEPVz7ICUUuRnHUNvP0n+kIGdg1I8RiHIpkZbhwfAfvC2UbxEdDWllYk+uI8lsgwk8mzpVcHx02LeTeKxOcTQn5f5dTC7k028RoU28ZY1laJIrGI2cRHytAVCRL0i56R4RyPnE3Z/c/Kg+0dVdqf925vrCRXILwYycqB26HNm5RS5RlYdSEiNwFXAv0icgz4MDpLKwzcZqmae6wMrSuAj4pIDp1a/C6l1HjVE69E1FIkIrDpGfbvJe08yhSJKrgjEvM6bUQkhYIesdsfDxen2x0e0/cpW/s6ePLUtK1IXKT/AoSDPneWSYm1NQqbn2U/1zkIm54JO3dXPTQU8PH1dz6HM9fG4d5uXWOSz5VWuBtFopS2tlZZ9Xo9GCLx+mwtf7glkl+IyAVKqUfdnlgp9foqmz9fY99voavmVx+Uqq1IylFibRlF4jiuUTEi2NZWGwXaJ5NZcgVFfzxcVCSHx/Qd/Rn9WpGYFvMRlzUHz9zay0Wb3HweZvxwSqf/OhVDIAxvv73u4ZdusewvQ+KpSdvWCnfZiiQ1oXubeURShGdtrRy4JZLnA78rIk+hYxkCKKsexMN8YIjBzUwQo0gKBX2caZFi4EqRWHfgbaRITA1Jf2e4mAl1eFwrkjN6dZbVSCKNiJ6q5wZf+N1nuntxo0ge+6ZOiZ5rAoKxwVITdurv2nN1r7R8rnoNySpHUZG4iHt5aG+4JZKXtnQVqxnlY3brwR/UfZ5MdXuZIkkHu2hIR34rrtJGwV1jW/V3hOiwrK2nLUWytc+OkUQCfsSZOrsQMIrk7r/XKb4XvnZu56mmSNaeA8d+pS0tM3OkWoxklSLgxUhWDNym/x4GEJF1gIsrngfXyDlIoRH8IcifLiUfhyI5mQ6ztdE5Ln0TbHl2WwV3TZ+t/s4wAava+YhFJFssIplO5VgTa0FTQ6NI/CF4xWfB7/beqgxFIpmwicSovsQpB5F4isQg5NWRrBi4Tf+9AfgEsAEYBs4A9gC7Wre0VYJmFUk+U0o+juNGc9HGRLL1efpfG6FobcXD5Aq6gv3psRmCfmGw2ybKllxwTIrv1X82v5b6Jj5VokjO1Y/TQ/aAso7VV3hYCwGvjmTFwO1t6V8AlwP7lFLbgKuBn7dsVasJRVJwQSSBsCaSbNI+RoScT6uZoUx7zF5vFqOJNH6f0BMN0mXFSKZTOXo7Qvh9UszkaokFMnA+vPteeM575neecmtL/HYdUOKUoyFkX/XjVyG89N+VA7dEklVKjQE+EfEppe4ALm7hulYPjCLxuygONFlbZXaYIZLjqeVJJGOJDL0dIXw+IRzwFe2t3g79fgyRhFtxwRGBdeeWti2ZC5xEkprQv5uOBNND2tqK9s7dOluB8LK2Vg7c/lVPWO1RfgZ8VUSG0cWJHuaLZhSJaZFSZocZIjk8szwHI+liRP0eRITOSIDTs1n6OjS5dkYCnJpq8+yeUIeePpm0YiTRNTpDLtZnKxIvPlICr45k5cDt/8zdQBL4H8APgYPAy1q1qFWFIim4DbZXKpKM6MdDiUoiuWv/SLF3VbtixOqzZWBaj/RaRGJ+d1uMuCQQsavbDZEAxNfrokSPSCrgWVsrB25H7c4opfJADPge8BVgZTRMXGo0FWwPVVUkNpEEyOXt+WMz6Rxv/sKv+Mc7DizokhcaYw5FAtAZ1oTYFzeKRP/e1ooEqhNJ54BFJCNe6m8ZzuiLEQv52dzbHlM6PcwdbkftvlNEhoBH0I0U72e1NFRsNZpK/w3q/cvssLQ1Y2yiEOXUVKq4+0w6R0HB7U8MVZyqnTCTzhUr2sFWIEVry4qRtH2aaKTbTv91KpLEkO6F5imSEpw90MkTH72WjT3Rxjt7aGu4jZH8IbBLKVV3mJWHOaApRWJlbeVM1pYmnzQhUipImhDHTyfZtEbf4SWtOecHR2Z4enSGrf0dVU+71EhlCyUkYYijPNje9haImUmSnLAr3Y0iUflVOXPEw+qAW6/gIDDbcC8PzaOGIlFK8Ylb93J03PGx+61ZIhlrm0U+SUJMiyaJY6eTxd0NkQDcvqc9VYlSinQuTzhg/yl2lsVIzO8tydpaSES6db+u1GSpIlHW9+BZWx5WKNwSyZ+gGzd+TkQ+af61cmGrBjUUych0mk/95AC3Om0p084jbc0Vs8hnkjhTAV2fcHzCQSQZm0h+vGd4gRe+MMgVFAVFCZEUra14WbB9OcRIJo8ByhEjcQzS8qwtDysUbq2tzwE/AR4FCg329dAMctWbNho1kXKoimKtSdoaHGmRz42Rt7CxA/oJc/x0JZFcuqWHXz89zmQyS3e0vVKEzfsrsbas4Lqd/hus2KctEe2x+6B5ROJhFcEtkeSUUh9o6UpWK2ooEkMk6RIiMYrEEIkmnxO5bro7uti4JlmqSKxjr79wAw8cmeDnB0a57oLBFryJuSOd0/clTkXSEzVZW/r9FYPt7Zz+C6Xdl4vW1oC9zSMSDysUbonkDhF5Bzr1N202rqrhU61CjRiJURPJqorEWFuR4j6xoJ9Na6I8ccIec2+OvXCTvsCdcJBMu8AokrCDJF592Wa29ncU1ZOJkURdziJZMjiJxPTeKlEkXozEw8qEWyL5bevxTxzbFLB9YZezCpFLgS8IvtK7bdvacjiJhmwMkVjEMpvJEwv56e0IcdvjQxQKCp9PimS0rjOC3yecns209r3MAUVF4oh/9HaEeMku+wJsx0jaXZE4BmkZRRKMQrhbZ9qFu5ZmXR48tBgNiUREfMAblFJek8ZWwIzMLUOqaozEEWy3GjaCVi/RUID1XWEy+QLjs3psrTk2FvbTEw0yPtN+Fe7prLG2apNEMUbS9tZWFSIBnQKcmZl/Py8PHtoUDb0CpVQB+PhcTi4iXxCRYRF5zLGtV0RuE5H91uMaa7tY2WAHROQREbl0Lq+57GBG5pYhmdEX2FTOoUic1pZp2JgvkMkXiIX8dFlWUCKl26DNWookGvSzpiPERBsqklTOsrbqZGQNdIXx+4SB7jYfhVMSI3GQSvdm6Nqw+Ovx4GGR4NZ0vlVEXiXNj6f7InBt2bY/Bn6slNoB/Nj6HfQUxh3Wv3cAn2nytZYnaiiShllbAV0NPGtUR8hfLNxLpHMl54gE/ayJBRmfaT8iMYqkntoY7I5y9wev4oodbR5jMEQSitvqEeA3Pw67P700a/LgYRHgNkbyAaADyItIEntme13TVyn1MxHZWrZ5N3Cl9fOXgDuBD1rbv6yUUsA9ItIjIoNKqZMu17g8UUuR1LO2UlPFY0wcJBryF2MJ0ymbSMIBH36fsCYW4vBY+9WUpl0oEqBkwFXbwqgQp60F0OuFEj2sbLht2tiplPIppYJKqS7r97lGDgcMOViPpm/ERuCoY79j1rYSiMg7ROQ+EblvZGRkjktoI+RS1WMkGZP+67S2HMF26xinfWWaHRpFksrki9Pn1sRCjLejteVCkSwbmGC609by4GEVwHU+pYjcICIft/5d34K1VLPNKjoMK6VuVEpdppS6bO3aFZCXn0vXVyS5Gum/1jGzGU0asZC/mCabSGeL5zD9qUyMRAu+9oFbRbIsEIxogi9XJB48rHC47f7718D7gCesf++zts0FQyIyaJ13ED0DHrQC2ezYbxNwYo6vsXxQQ5EYInG2OSlaW4WsXUNStLYCFdbWbMZBJLEg2bwqqpV2QbWCxGWNSI9HJB5WHdz+770OeLFS6gtKqS+gA+jXzfE1vwu82fr5zcB3HNvfZGVvXQ5Mrvj4CNRWJJk6igQciqQy2G6IJJXNF2sv1ljtRiZm2ysFOF2lRcqyxnP/O1z024338+BhBaGZAdI9gKlk7663o4GI3IQOrPeLyDHgw8BfA98QkbcBR4BXW7v/F5qcDqA7Db+libUtX+RSehxrGVLVChJLiKQyRhIO+Aj6pSRrK+aIkQCMz2TaapDQilMkz/39pV6BBw+LDrdE8lfAgyJyBzqWcQWlVe5VoZR6fY2nrq6yrwLe43I9KweNYiTVsrbAztrK2jESESEeDhTrSJKZPB3F2R762FrV7cdOz3Lf06d5+SUV+Q0thU0kK0SRePCwClH3NlBEnmf9+B/A5dbjfwDPUUp9rcVrWx2oFSOplrXlJJxijEQ/HwtpwohHAg5FYg+MMoqkFpF89d4jvP/rDxWD94uFVDaPTyDo96q+PXhYrmjkJ5iZI79USp1USn1XKfUdpdSpVi9s1aCBIsnkC+QLVqZVVWtLX/hNmm9nOGjXkWRyjmC7RSQ12qQMTeouxENT6arPtwrpXIFwQKspDx48LE80srayIvKvwKZqg6yUUu9tzbJWEWrVkTgsrXQurxVHNWvLEWwHo0gq03+7okF8UluRDE9rAjk5mWTbIo7kTWXzKyP114OHVYxGRHI9cA3wIuD+1i9nFaKBIgFNFppIqiiSbJ6ATwj69cW4Mxzg1FSqeJxRKn6f0B0N1iSSoalUyeNiIZ0trIxiRA8eVjHqEolSalREbgY2KKW+tEhrWj1Qqk7TRptIio0bfdUViSELsBTJiEn/LZQ8t6YjVNPaMork1ORiW1ueIvHgYbnDTfffPPCyRVjL6kM+C6iqRJLKFghZKqNoc/l84LO43xEjiTmJxMraMl2Bo476jDWxUFVFksrmmUxqgllsRZLyFIkHD8sebtN/fyEi/wh8HZgxG5VSD7RkVasFNcbsgra2emJBhqfTZSnAYSjkSgoSTcYWaEUync4VrbFyIjl2urJx48i0rUJOTS6yteUpEg8elj3cEslzrcePOrYpdOzEw1xhjdk9OlUo6Q0D2rLa0huziMRZlBiELCUtUpxk0RkOkMkVigoj4rS2YkEeO15pbQ1Pa/II+KQYX1ks6Kwtj0g8eFjOcNv996oq/zwSmQPu3j/K9x62Wojl/397Zx4fVXku/u87k2WybyQhJEACokAghE02FRBF9F4VrLjUKtTtXpdP1f6sWnu1en9d1NraUmmtXdSqV6Eq/qzWWlDwWqqySEBZgxAgJGTfk8lkkuf3xzkzmQmTBTKTyYT3+/nMZ855z/acM+fMc57ned/nMRTJ6k+OedUKERG3RQKdaUQAd8D96Y+KaLC3ucvsunBVE6xsNPbnqWSSY4wMwF0TN7q6/J4zPC4Irq32oZMeRaM5Q+lr0sZ0pdQflVLvm/MTzRQnmlPk+U8OsXr9bmg44bZIWiTcbUFA52hv19gPX/m2ShqFw5VNNLd1CbabI9ld7ipPJZMYHYHD2eHVIwyg3FQeeVkJlDe0do5bGQC0RaLRhD59fYJfBD4AXPVCDwD3BkKgoU51UyvLWt6AX0+H2qMAtBJBk0dWXlePrSQzrclJri1zm5JaOy1dg+02b0XibZGEmzJ4B9zLG1oJsygmZMTT3iFUNg5czy3XgESNRhO69FWRDBORtUAHgIg4gfaeN9H4orrRwdnOQnA0wuZfAtBKuFd6d5fFkGhaJN6p5CPc25TUtpiJGTtDXXGmReKKe3i6jVz765oBuKy+lbS4SHcVwoEMuOsBiRpN6NPXJ7hJKZWCWWjKleY9YFINYaqbHeRQbMwc2gQYSsEzx5VLkST7cm2FdSqS0roWWhzeMQaXReIaF+Lp9kqO8Z1vq7zBTmq8jeHxRgB/IAPu2iLRaEKfviqS72LUCxmjlNoM/BnQ+bJPkWaHE9paGEU5znGXuttbJZzGVu+R7IA72O4rlXyrhFNSaz8p2N41RuLd/bcb15ZpkaQnGF2KBzLg3mrWlddoNKFLX5/gPcA6YCtQBvweI06iOQWqGh2MVSVYlNB4zlVw1kWAYV00+XBtuYPtPnpttRLOcbdr6+ReW76C7UnduLbKG+ykx0cyLCbS6AI8kK4tZ4futaXRhDh9VSR/BsYDPwF+DYwDXg6UUEOVmmYH49RxAOpiz4IL/4s9ahxHJc1nsD0hOhylunb/7Qy2H6poRMTbfRXXJdju+SedEHWyRdLqbKemuY20OBsWiyI93jZgri0RwaF7bWk0IU9fBySeIyJTPOY3KqV2BkKgoUxVk4OzLcW0iZU6WxaMSOM6+TH1OGnydG21eVc9dOfaAsQagQLaVDj1Zrr4aA9lERlmIcyiqGg8OUYSZrWQHh/JkSp3cgK3wkmLM9xa6fGRA2aRuLo5a4tEowlt+voquMMMsAOglJoFbA6MSEOXmiYH41Qxh2U4DU4LHR1Cg2mJNHkE212urKgIK7Zwq5dry2nq/rSkzmrHnr22lFLE2ozR7QC2Lm/7M0Yns7Woxj3vCsqnm4H24QkDZ5G4inZpi0SjCW36+gTPwsi3VaSUKgI+BeYrpb5USu06lQMqpc5RShV4fOqVUvcqpR5TSh33aL/sFM9l0FPdZLi2DkgWja1OGlqduAaZN/pwbUVHWLGFWb26/7ZhuKdyhnfWefe0OqAz4B5htRBm9f6JZ2Yncby2xZ1zyzUYMdVtkdjcRa4CTavZG013/9VoQpu+uraW+OuAIrIfyAdQSlmB4xiB/G8Dz4jI0/461mCjrr6eUaqctzvmMdLupN5jNHuzj2B7VLiVqAirl2urVaxEA2eNSIHdRjfi6G4USVcFAzAzJxmArUXVZCVFn2SRZCTYaHK0U9PkICkm4qTt/YmrN5rO/qvRhDZ9zbV1pKdPP46/CPi6n/sIGcJrDmJRwoEOwyLxTIvS6CNGYnPFSDxcW/YO40/37Mxh7rauCiPe7LkV5SP2MH54PHGRYWw5bLi3yutbsVoUKabSODfHsHQ+2B34asraItFohgbBfoKvA17zmL9bKbVLKfUnpVSSrw2UUrcrpbYppbZVVFQMjJR+IrquEMDt2nJZJErh1WvL7mhHKSN20DVGYhcrHaLISI53jzPpqjBcgxJ9WSRWi2JGdhJbi6oBY8zIsNgILBajZvqUrATGpMbw1hfH/XXa3eIKtusBiRpNaBM0RaKUigCuAP5iNv0WGIvh9ioFfu5rOxF5XkRmiMiM1NTUAZHVXyQ3HcKJleMqw1AkdkORpMVFnjSyPSrcilIKW7jFHZQGaGm30ko4STGRjDBTmngG26HTtdVdb6iZOckcLG+kuslBeUOr260FRrD+qqmZbCmq5lj1ybVL/IndbXkF+31Go9H0h2A+wZcCX4hIGYCIlIlIu4h0YAx4PDeIsgWE9NYiKiKyiLTZaPJwbWUkRJ2Ua8tlZdjCrV4pUnYlLOQ37VeSEBXOiERDAZwUI3FZJN38QZ+b3RknKau3u7v+ulg6NROAdTsCa5Voi0SjGRoEU5Fcj4dbSymV4bFsGfDVgEsUYEa0F1MdnUNMhFEOt77FUB4jEm3e40gcnaO9bWHerq3dYbm8HHENVotiRKJhkXR1YbkSN3a1VFxMzkog3hbGk+/v43htC6lx3hUas5KimZWTzLodx0+qXeJP3DES3f1XowlpgvIEK6WigYuBtzyan/LoTrwQuC8YsgWKNmc7aVJFa3QGcbYwd7DdalGkxdlOGkfiUg62cItXrq2aZoc71cnZ6XHERFjdriwXvbm2IsOs/HHlTCoaW2mwO0mPP7lm/DemZXG4sokdx2r7d+I94O61pQckajQhTVAUiYg0i0iKiNR5tN0oIpNFJE9ErhCR0mDIFijqaquJUa1IbAYxkWHuGEm8LYzYyDCaWp3ut39P11ZUhNWrEFVtc5s7yH7dzJFsvH/BSX/EPQXbXczMTub122dzTnocM0Ynn7T80snDiQyz8E5BSf9O3Ac/fm8Pb31RrC0SjWaIoJ/gAaKh3ChipRJGuBVHXUsb8VHhxESG0SGdb+ieddgju7i2qpsc7vTyYVYLafE2uhLn7v7b88+bOyKBD+67gPPGDTtpWZwtnOmjk9h+pMbHlv3j9a3HeG9XaefIdh1s12hCGv0EDxAtVccAiEjKJDYyjAaz+29CVDgxkYbScAXcW9rasUV0Bts9e23VNjvcBaq6wz0gsZ8uo/yRiewtrffOPtxPmh1OGuxOSursnb22dLBdowlptCIZINpqzFHoKSO9LRJbODFmUNzVBdje1u62JmzhFhztHe466jXNbe66It0R53Zt9TVxgW/yRybi7BC+Ou6/Gmbl9cZI+tK6ls5eW9oi0WhCGv0EDxBSb8QaYlOzjBiJ3Um93WlaJMYfvqdF4tn9F4weTva2dlra2ntNXeI3i2RUIgAFfgy4u4pm1Ta3UWPWRdHdfzWa0KZ/r6yaPmNpPEGNxJKUkECsrYImRzu1zQ7io8Lcri1XF+AWh0evLTMQbW/rcAenk3pzbbktkv69J6TF2chMjPJrz60yM7cXQFFlE+FWhdUcVa/RaEITbZEMEBHNJyhXKYRbLe5xHpWNDnewHTpTybe0ddZhd33b29qpaTLe4HtzbSVHR2C1qF5jKX0hf2QiBUf9p0jKPVLUF1U1aWtEoxkCaEUyQES3llNjNXpHxXiM+4i3hbtdUa58W/Yu3X/BUC61zUZlw94URFJMBG/fOY8r80f0W+6poxI5XtviLoDVX8o9LZKqJp0eRaMZAuineICIb6ukIcLIDeZyPQFeMZKmVidt7R20tYtX918wlEu1qUiS+5DefXJWgl/e9vNH+jdOUlZvJyPB6LJsb+vQFolGMwTQMZKBoL2N+I5aWiLTAIiN7PzzjI8KJyaiM0bisko8R7aD8afrCk735tryJ5MyEwizKN7ZWcLbBcc5XtPC2v+YQ8RpDiIsq7czMimatvYOKhsdejCiRjME0E+xv9j6R/jgB76XNZzAgtAWnQ5AbGSnIuhqkRw1M+6OTI4GPHpttbVT29Q315Y/sYVbGZ8Rx193lvCP3ScoOFbLlsPVp72/8vpWUuMjyTAzF0fq9CgaTcijFYm/2PcefPWmz0VSb2TRbY8z8lLGeFoktjDCrRYiwiw0OpwUVRmKJDslBvAItjsN11ZMhPW0rYHT5cEl43n4svH888ELiQyzsGFv2Wnvq6zeTnqcze3e0haJRhP66KfYXzRXQlMF+MiW21JtDEa0xhvB77guFglATISV5tZ2iiqbABid4rJIOl1btc1tAS9/64vzx6Vy+wVjSY+3Me+sYXy4r+y0sgI3tjppcrSTHh/pzlysFYlGE/rop9hfNFVChxPsJwel7VWGIolIygK6WCQuRWKOdi+qaiIjweaVRh7M7r8emX+DxaIJaRyrbuFgeeMpb+sajJge32mR6My/Gk3ooxWJPxAxFAl0fnvQVnOcVgknNskMttu8u/+CMRq9sdVJUWWT260F3t1/959oYJQZOwkWF443zmHD3nKaHU5WvrCFZz8qpKOjdwvFpUjS4iPJ0BaJRjNk0L22/EFrA7Sb4yOaKmDYOK/FUl9CmSSSHGvU/YgMsxJuVYRZLO54R0xkGE1mjOSS3HT3ti6LZP+JBkrr7MwemzIAJ9Q9GQlR5I6I56N9ZZyoa2HT/go27a9gV3EdP79mijvzsC9cebbS421EWF25xLRF0hfa2tooLi7Gbrf3vrJGY2Kz2cjKyiI8PLA9PbUi8QdNFZ3TjeXey0SIqDvMUZLJ8IhvxEaGeY2hiI6wcry2heomh5dF4kpouHG/sd+5QVYkAIsmpPPrjwrZWlTDyrnZjEqO5sd/28ulv/qEp5dPYfYY3zJ6urZs7nEy2iLpC8XFxcTFxZGdnY1SOqWMpndEhKqqKoqLi8nJyQnosfRT7A+aqzqnPZUKwI6XSa7bzQftM7wGEsZEhhEf1anHYyPDOOwOtHsokjALSsGx6hbS4yMZM6xzWbC4aEIaIpCdEs2DS8Zz83k5rP2P2Vgtiuue/4xfrD/gc7uy+laizYqO6XGRWJTO/NtX7HY7KSkpWolo+oxSipSUlAGxYoP2FCuliszSugVKqW1mW7JSar1SqtD8TgqWfKeEp/LwjJFU7If3H+Rw/AxeVf9GtEfFwtjIMHePLTAUi6sjVI6HslBKud/a54wZHH8kk0YkcMeCsay+YZo7hjN9dDLv33M+l00ezq8/KqSq8eSUKuUNdtLNQlxhVgs3zclmwdlpAyp7KDMYfntNaDFQ90ywXwcXiki+iMww5x8CPhSRccCH5vzgEA5DrAAAHN5JREFUx608VKdSabPDG7dAeBQvD3+YpBib14+6eGI6iyZ0xkJiPJRM14C6yw00d+zJlQyDgcWieHDJeHJHJHi1R0eEccf8sxCBjfs7levnh6q4b00B6/eUuXtrATx2RS4XTUxHo9GENoMtRnIlsMCcfgnYBDwYLGH6QmVjK60lx8gESBzVqUg2/QTKvoRvrqXoX/Ekx3ibl99dfI7XvGt0e0aC7aRa60bAvY05gyA+0huTMuNJj4/kw71lXD09i+1Hqrn2+c+It4Vx1bQsbp6XHWwRNRqNnwmmRSLAP5RS25VSt5tt6SJSCmB+D3q/x282fs2HW79CIuIgYWSndbL/73DWxXD2JVQ1OXpNtOhSJK6BiJ5ERVjJSopyp00ZzCilWDQhnf89UEGrs51fbigkJSaCf31/ET+9ajLj0uOCLaImBFmwYAHbtm0D4LLLLqO21n+lDXqjqKiISZMm+X2/P/nJT/y+z2ARTEUyT0SmAZcCdymlLujLRkqp25VS25RS2yoqKnrfIMAcrW4mXupoj06B2FTDIunogNojkGpYHaW1Le7YQHe4Usl79thyMe+sFK6bOdL/wgeIiyak0eRo53cfH+KTwkpuu2CM+/w0Zx5Op9Ov+/vb3/5GYmKiX/cZDLQi8QMiUmJ+lwPrgHOBMqVUBoD5Xe5ju+dFZIaIzEhNTR1IkX1SUttCCvW0RiRDTCo0lUNjGTjtkJxDi6Od8oZWRvdiTbgC8dk+emX9aOlk7r5w3Entg5W5Y4dhC7fwzIYDJEWHc+Ps0cEWSdNPioqKmDBhArfddhu5ubksXryYlpYWCgoKmD17Nnl5eSxbtoyamhrAsCAefvhh5s+fz69+9StWrlzJHXfcwcKFCxkzZgwff/wxN998MxMmTGDlypXu49xxxx3MmDGD3NxcfvjDH/qUJTs7m8rKSp577jny8/PJz88nJyeHhQsXAvCPf/yDOXPmMG3aNJYvX05jY/dZGP77v/+bmTNnMmnSJG6//XZ36p/t27czZcoU5syZw+rVq93rz5o1i927d7vnFyxYwPbt29myZQtz585l6tSpzJ07l/379wPw4osvctVVV7FkyRLGjRvHAw88AMBDDz1ES0sL+fn53HDDDQAsXbqU6dOnk5uby/PPP+8+xh//+EfOPvtsFixYwG233cbdd98NQEVFBd/4xjeYOXMmM2fOZPPmzX37MQOBiAz4B4gB4jym/wUsAX4GPGS2PwQ81dN+pk+fLkHjyGciTodMefwD2f3IZDnx3FKRTU+K/DBe5OuNxnfhetlXWi+jH3xX/l/B8R5397ddJTL6wXfl/S9LBkT8QHPLi1tl9IPvyrMfFQZblCHBnj17gnr8w4cPi9VqlR07doiIyPLly+Xll1+WyZMny6ZNm0RE5JFHHpF77rlHRETmz58vd9xxh3v7FStWyLXXXisdHR3y9ttvS1xcnOzatUva29tl2rRp7v1WVVWJiIjT6ZT58+fLzp073fvbunWriIiMHj1aKioq3Pt2OBxy3nnnyTvvvCMVFRVy/vnnS2Njo4iIPPHEE/L44493e16u44mIfOtb35J33nlHRMTrvO6//37Jzc0VEZFf/OIX8uijj4qISElJiYwbN05EROrq6qStrU1ERNavXy9XXXWViIi88MILkpOTI7W1tdLS0iKjRo2So0ePiohITEyMT1mam5slNzdXKisr5fjx4zJ69Gipqqpyn+ddd90lIiLXX3+9fPLJJyIicuTIERk/frzPc/R17wDbxI//6cHyN6QD68xeTGHA/4jI35VSW4G1SqlbgKPA8iDJ1zOVB+FPi2m99Blqm9NJiayjRiWQHmP2qio2fLkk5XDkhDk2pBeLJHdEAhMy4pk2KjR6PPfG9eeOpKLBzk1ztDUyVMjJySE/Px+A6dOn8/XXX1NbW8v8+fMBWLFiBcuXdz6y1157rdf2l19+OUopJk+eTHp6OpMnTwYgNzeXoqIi8vPzWbt2Lc8//zxOp5PS0lL27NlDXl5ej3Ldc889XHjhhVx++eW8++677Nmzh3nz5gHgcDiYM2dOt9tu3LiRp556iubmZqqrq8nNzeWCCy7wOq8bb7yR999/H4BrrrmGiy++mMcff5y1a9e6z7euro4VK1ZQWFiIUoq2tjb3MRYtWkRCgtHDceLEiRw5coSRI092Va9atYp169YBcOzYMQoLCzlx4gTz588nOTkZgOXLl3PggDFOa8OGDezZs8e9fX19PQ0NDcTFDXwcMiiKREQOAVN8tFcBiwZeom5ob4M134ILvgdZMzrbj28HwF70OXA5yTRQ2BFnuLbAUCTKAgkjObrHSNjoK4juyaiUaN6/5/xAnEVQWDTBu3uzJvSJjIx0T1ut1l4D3jEx3m5a1/YWi8VrXxaLBafTyeHDh3n66afZunUrSUlJrFy5stfBdC+++CJHjhzh2WefBQwPy8UXX8xrr73W6/nY7XbuvPNOtm3bxsiRI3nsscew2+2ISLfjLzIzM0lJSWHXrl2sWbOG3/3udwA88sgjLFy4kHXr1lFUVMSCBQtOOm8wrpuvmNGmTZvYsGEDn376KdHR0SxYsMAtS3d0dHTw6aefEhUV1eu5BppgjyMZ1DSXfQ0H/k77gfXeC0p3AmA5sZN4mghX7ZS0xXookq0QnwVhERypaibeFjagxag0moEgISGBpKQkPvnkEwBefvll91v86VBfX09MTAwJCQmUlZW5rYDu2L59O08//TSvvPIKFovxVzZ79mw2b97MwYMHAWhubna/wXfFpaSGDRtGY2Mjb7zxBgCJiYkkJCTwz3/+E4BXX33Va7vrrruOp556irq6OrdVVVdXR2ZmJmAot74QHh7utlzq6upISkoiOjqaffv28dlnnwFw7rnn8vHHH1NTU4PT6eTNNztrHi1evNitQAEKCgr6dNxAoBVJDxSYZmNVaZH3AlORxNQWMkIZ1QKPtcZ0KpLmSkjOBuBIdbNXyhONZijx0ksv8b3vfY+8vDwKCgp49NFHT3tfU6ZMYerUqeTm5nLzzTe73VPd8eyzz1JdXc3ChQvJz8/n1ltvJTU1lRdffJHrr7+evLw8Zs+ezb59+3xun5iYyG233cbkyZNZunQpM2fOdC974YUXuOuuu5gzZ85Jb/xXX301r7/+Otdcc4277YEHHuD73/8+8+bNo729vU/ne/vtt5OXl8cNN9zAkiVLcDqd5OXl8cgjjzB79mzAsIAefvhhZs2axUUXXcTEiRPdbrJVq1axbds28vLymDhxIs8991yfjhsIVE+m02BnxowZ4upbHgg2rf0VC/Y8Smna+WTc+a7R2NEBT4yCyDhoKOFHzhv5r7CXucvyKKsfuNVYBjDtJrji1yz42UZyMxNY/c1pAZNTM/TZu3cvEyZMCLYYmiDQ2NhIbGwsTqeTZcuWcfPNN7Ns2bI+b+/r3lFKbZfOjCL9RlskPaDqSwEIayztbKw5DI4GmPotABZH7ALgUEsUdksMWE1/aFIOzvYOimtaeg20azQaTXc89thj5OfnM2nSJHJycli6dGmwRToJPUqsB8KaDAUSbfcYzlKyw/iecDkNn/yWqR1Gn/IqiedEfSvZMalQXwxJ2ZTU2nF2SK+Bdo1GE1iWLVvG4cOHvdqefPJJLrnkkiBJ1HeefvrpYIvQK1qR9ECU/QQAMR314GiGiGgjPmKNgLQJ7FVjOFeMeEkNcZTUtZAdM8ytSI5UG11/RyXrGIlGE0xc3Wo1gUG7tnog3uGRgqXBdG+V7oT0XNpVGF+0GWMk2iMTaCOMklp7Z8A9OYcjVc1A711/NRqNJpTRiqQHktsrONQxHAB79TGjNnvpTsiYQmVjKzvbjapjylQeJbUtEJcOUUkQlcTR6mYiwiwM7yXPlkaj0YQyWpF0Q7ujhWTqORgxHoDq0iIjEaO9FjKmcLy2hS9lDACW2FSGxUZSUttC86z7WHf2k1Q3OThS1cTIpCgsFl2QSKPRDF20IumGuvKjADQMM1JCNFYchWNbjIWZ0ympbaFYhuG0JUHMMDITbZTU2XnlgOK+z2NZ8actHChr1GNINEMCq9XqTpCYn59PUVFRt+sGKu26ZvCig+3d0FBWRDKQOHIidaXRtNUUw+FysCVA+iRKCosAhePfVxOWnEnGh+0Uljfw+tZmMhOj2Ftaj7NDmH928DMUazT9JSoqKqgjpzWDG61IuqGl8hgAqZljKCcFS0MpNB2F0eeBxUpJrZ04WxjRk4zugyMS9/D33UYvr59dnUdEmIV71xRwznBdyEnjXx7/6272lNT7dZ8TR8Tzw8tzT2mboqIibrzxRpqajN6Jzz77LHPnzvVaZ/fu3Xz729/G4XDQ0dHBm2++ybhx43jllVdYtWoVDoeDWbNm8Zvf/Aar1errMJoQQCuSbnDWGskW49NGURWRypjGr6C9Gmb9JwDHa1vITOxMnTAi0Qiox9nC+Pe8EURFWJk2KsmrRrlGE6q4ameAkQV43bp1pKWlsX79emw2G4WFhVx//fV0zTTx3HPPcc8993DDDTfgcDhob29n7969rFmzhs2bNxMeHs6dd97Jq6++yk033RSMU9P4Aa1IXLQ2QOUBqC8xSuTWl1Av0QxLSeF41HCS6r8A4GcH0nhr44eUN7SywMNt5VIqS/Mz3TXXQ6E0rib0OFXLwR/4cm21tbVx9913U1BQgNVq9Zkccc6cOfz4xz+muLiYq666inHjxvHhhx+yfft2d26rlpYW0tIGfVVtTQ9oRQLQUAar8qHNGPfB/AcJayqljGTOirDSEZsB9dAclsBv9kRw6eREMhKiuGLKCPcupoxMZPzwOFbMzQ7OOWg0A8wzzzxDeno6O3fupKOjA5vtZOv7m9/8JrNmzeK9997jkksu4Q9/+AMiwooVK/jpT38aBKk1gUArEoAj/zSUyGVPw4EP4PPnSOqI57A1lXFKEZ6UCSWwqfUcrpkxmievPrnQzojEKP5+b5/Kzms0Q4K6ujqysrKwWCy89NJLPrPeHjp0iDFjxvCd73yHQ4cOsWvXLhYvXsyVV17JfffdR1paGtXV1TQ0NDB6tC6CFqro7r9gdOsNi4LpK2Hh98FeR7rjGA0Rhrkdm2rc4Idjp/H4lQPvVtBoBiN33nknL730ErNnz+bAgQMnFbICWLNmDZMmTSI/P599+/Zx0003MXHiRH70ox+xePFi8vLyuPjiiyktLfVxBE2ooNPIAzy/EMKj4dvvGfN/XgqHNvLXpBVcfs8qmuqr+epPdzH8G08yeuSo/h9PozlFdBp5zekyJNPIK6VGKqU2KqX2KqV2K6XuMdsfU0odV0oVmJ/LBkQgRzNyYhd1w6Z2tl1wPwD2WKOuckx8MrPufU0rEY1Go/FBMFxbTuD/iMgEYDZwl1JqornsGRHJNz9/GxBpSnagOpx897NI9p9oAKBt5FyuaP2/lI4cGF2m0Wg0ocyAKxIRKRWRL8zpBmAvkDnQcrho+vpfAHzRfhbr9xgDCqsaHeySsaQkxAZLLI1GowkZghpsV0plA1OBz82mu5VSu5RSf1JKJXWzze1KqW1KqW0VFRW+Vjklju7cxCHJICYpnQ/3GQWsKhtbARgWG9nv/Ws0Gs1QJ2iKRCkVC7wJ3Csi9cBvgbFAPlAK/NzXdiLyvIjMEJEZqan9y2P1dXkDaXU7qRs2lWtmjKTgWC2Vja3sM11cqXFakWg0Gk1vBEWRKKXCMZTIqyLyFoCIlIlIu4h0AL8Hzg20HAUF20lRDWTnL+TC8WmIwBvbi3ni/b1MyIhncmZCoEXQaDSakGfAByQqpRTwR2CviPzCoz1DRFydyZcBXwValrZDnwCQNH4BicPiGR5v48m/7yPcYuHVW/MJt+phNhpNVVUVixYtAuDEiRNYrVZc3oAtW7YQERERTPE0g4BgjGyfB9wIfKmUciXveRi4XimVDwhQBPxHoAVJrtxGnTWJhGHjUEpx4YQ0/ufzozyw5BydtVejMUlJSXHn2XrssceIjY3l/vvv91pHRBARLBb98nUmMuCKRET+CfgqGTgw3X1NaptayW37koq06SQoQ5zbzx/DqORobp6XM5CiaDSnxvsPwYkv/bvP4ZPh0idOaZODBw+ydOlSzjvvPD7//HPefvttpkyZQm1tLQCvv/46GzZs4A9/+ANlZWXccccdHD16FIvFwqpVq5g9e7Z/z0ETNM7Y14cD+3eTqapQ2fPcbdnDYvjP+WN1aVyNpo/s2bOHW265hR07dpCZ2X0v/u985zs88MADbNu2jbVr13LrrbcOoJSaQHPGJm2s2/cxAOmTFwVZEo3mFDlFyyGQjB071p0Ovic2bNjA/v373fM1NTW0tLQQFRXVw1aaUOGMVSRRJZ9STxzxWZODLYpGE7J4Jmq0WCx45u6z2+3uaRHRgfkhzBnp2hIRshsLOBKXDzo4qNH4BYvFQlJSEoWFhXR0dLBu3Tr3sosuuojVq1e753X996HFGfkveuLY12RRRuuIWcEWRaMZUjz55JMsWbKERYsWkZWV5W5fvXo1mzdvJi8vj4kTJ/L73/8+iFJq/M0ZmUb+6y8/o+2d+wi7/OeclTc3AJJpNP5Fp5HXnC4DkUb+jIyRjJ08GyZ/GmwxNBqNZkhwRrq2NBqNRuM/tCLRaEKEUHZDa4LDQN0zWpFoNCGAzWajqqpKKxNNnxERqqqqsNlsAT/WGRkj0WhCjaysLIqLi/FHDR7NmYPNZvPqPRcotCLRaEKA8PBwcnJ0DjjN4ES7tjQajUbTL7Qi0Wg0Gk2/0IpEo9FoNP0ipEe2K6UqgCOnsekwoNLP4gSaUJQZQlPuUJQZQlPuUJQZQl/u0SKS6q+dhrQiOV2UUtv8mR5gIAhFmSE05Q5FmSE05Q5FmUHL3RXt2tJoNBpNv9CKRKPRaDT94kxVJM8HW4DTIBRlhtCUOxRlhtCUOxRlBi23F2dkjESj0Wg0/uNMtUg0Go1G4ye0ItFoNBpNvzijFIlSaolSar9S6qBS6qFBIE+RUupLpVSBUmqb2ZaslFqvlCo0v5PMdqWUWmXKvkspNc1jPyvM9QuVUisCIOeflFLlSqmvPNr8JqdSarp5HQ6a26oAyv2YUuq4ec0LlFKXeSz7vinDfqXUJR7tPu8bpVSOUupz83zWKKUi/CDzSKXURqXUXqXUbqXUPWb7oL3ePcg82K+1TSm1RSm105T78Z6OpZSKNOcPmsuzT/d8AiT3i0qpwx7XO99sD/w9IiJnxAewAl8DY4AIYCcwMcgyFQHDurQ9BTxkTj8EPGlOXwa8DyhgNvC52Z4MHDK/k8zpJD/LeQEwDfgqEHICW4A55jbvA5cGUO7HgPt9rDvRvCcigRzzXrH2dN8Aa4HrzOnngDv8IHMGMM2cjgMOmLIN2uvdg8yD/VorINacDgc+N6+hz2MBdwLPmdPXAWtO93wCJPeLwNU+1g/4PXImWSTnAgdF5JCIOIDXgSuDLJMvrgReMqdfApZ6tP9ZDD4DEpVSGcAlwHoRqRaRGmA9sMSfAonI/wLVgZDTXBYvIp+KcQf/2WNfgZC7O64EXheRVhE5DBzEuGd83jfmG9qFwBvm9p7XoD8yl4rIF+Z0A7AXyGQQX+8eZO6OwXKtRUQazdlw8yM9HMvzN3gDWGTKdkrnE0C5uyPg98iZpEgygWMe88X0fLMPBAL8Qym1XSl1u9mWLiKlYDygQJrZ3p38wTovf8mZaU53bQ8kd5sm/p9cLqJe5PPVngLUioizS7vfMF0nUzHeOEPieneRGQb5tVZKWZVSBUA5xh/p1z0cyy2fubzOlG3An82ucouI63r/2LzezyilIrvK3Uf5TvkeOZMUiS8fX7D7Ps8TkWnApcBdSqkLeli3O/kH23mdqpwDLf9vgbFAPlAK/NxsH1RyK6VigTeBe0WkvqdVu5FjwOX2IfOgv9Yi0i4i+UAWhgUxoYdjDVq5lVKTgO8D44GZGO6qB83VAy73maRIioGRHvNZQEmQZAFARErM73JgHcaNXGaalpjf5ebq3ckfrPPyl5zF5nTX9oAgImXmQ9gB/B7jmp+O3JUYLoKwLu39RikVjvGH/KqIvGU2D+rr7UvmULjWLkSkFtiEEUPo7lhu+czlCRiu06A9mx5yLzFdjCIircALnP71PvV7pC/BnaHwwagGeQgjGOYKfOUGUZ4YIM5j+l8YsY2f4R1Ufcqc/je8A2ZbpDNgdhgjWJZkTicHQN5svIPWfpMT2Gqu6wrsXRZAuTM8pu/D8G0D5OIdMD2EESzt9r4B/oJ3UPZOP8irMHzSv+zSPmivdw8yD/ZrnQokmtNRwCfAv3d3LOAuvIPta0/3fAIkd4bH7/FL4ImBukf8+mcz2D8YvRcOYPhBfxBkWcaYN9ZOYLdLHgyf64dAofnt+mEVsNqU/Utghse+bsYI8B0Evh0AWV/DcE20Ybyt3OJPOYEZwFfmNs9iZlwIkNwvm3LtAt7B+8/uB6YM+/HopdLdfWP+hlvM8/kLEOkHmc/DcCPsAgrMz2WD+Xr3IPNgv9Z5wA5Tvq+AR3s6FmAz5w+ay8ec7vkESO6PzOv9FfAKnT27An6P6BQpGo1Go+kXZ1KMRKPRaDQBQCsSjUaj0fQLrUg0Go1G0y+0ItFoNBpNv9CKRKPRaDT9QisSjUaj0fQLrUg0IY1SKlEpdWcv62Qrpb7Zh31lK4+U8z6W53dJhX6Fv1KDd3O8pUqpiYHav0bjL7Qi0YQ6iRjpvXsiG+hVkfSBfIwBZgCIyDsi8oQf9tsdSzFSlGs0gxo9IFET0iilXKm592NkbwUjCaYAPxKRNUqpzzCS8R3GSAO+DmPUdYy5/t0i8i8zc+27IjLJx3EiMEb/RgHHgZ+a0zNE5G6l1ItAC0bSvNHAt4EVGDUdPheRleZ+FgOPY6TT+BpjNHGjUuoJ4ArACfwDeAt4FyPDbB3wDVOU1RgpMpqB20Rkn3lsO0aqjnTguyLyrlIqFyPnUgTGS+M3RKTw1K6wRtMH/DFkX3/0J1gfPHJpYfzZrsfIc5QOHMUourQAQ0G4tokGbOb0OGBb1311c6yVwLO+5jGKCr2OkY7iSqAemIzxB74dw5oZBvwvEGNu8yDwKEbOo/10vtgleuzzao/jfQiMM6dnAR95rPd381jjMNLB2IBfAzeY60QAUcH+vfRnaH5cGS41mqHAecBrItKOkS33Y4yU2l3TsIcDz5qlSNuBs/10/L+KiCilvgTKRORLAKXUbgwllYXhqtpsVi6NAD415bMDf1BKvYdhiXhhpmifC/zFo+pppMcqa8XIsluolDqEYRl9CvxAKZUFvCXaGtEECK1INEOJvtYevw8oA6ZgvMXb/XT8VvO7w2PaNR+GobTWi8j1XTdUSp0LLMLIKns3RpU+TywYBZfyuzl2Vx+1iMj/KKU+x8j++oFS6lYR+ehUTkij6Qs62K4JdRow6oSD4Ta61qwel4pRs31Ll3XAqCNRar7B34jhCjvVY50OnwHzlFJnASilopVSZ5vWRoKI/A24F8MN5nU8MQpFHVZKLTe3VUqpKR77Xq6UsiilxmJkr92vlBoDHBKRVRjZd/P6IbtG0y1akWhCGhGpwnAVfYUR2N6FkZr/I+ABETlhtjmVUjuVUvcBvwFWmEH4s4GmPh5uIzBRKVWglLr2NGStwIirvKaU2oWhWMZjKIt3zbaPMSwmMGIu31NK7TAVxA3ALUopV+kBz/rf+81t3wf+U0TswLXAV2ZJ1vEYNUM0Gr+je21pNCGO2WvrXRF5I9iyaM5MtEWi0Wg0mn6hLRKNpgtKqUuAJ7s0HxaRZcGQR6MZ7GhFotFoNJp+oV1bGo1Go+kXWpFoNBqNpl9oRaLRaDSafqEViUaj0Wj6xf8HeKpEnKj/srgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "pg_result_no_na[\"normalize_advantage\"] = False\n",
    "pg_result_na[\"normalize_advantage\"] = True\n",
    "pg_result = pd.concat([pg_result_no_na, pg_result_na])\n",
    "ax = sns.lineplot(\n",
    "    x=\"total_timesteps\", \n",
    "    y=\"performance\", \n",
    "    data=pg_result, hue=\"normalize_advantage\",\n",
    ")\n",
    "ax.set_title(\"Policy Gradient\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 3: Policy gradient with baseline\n",
    "\n",
    "(25 / 100 points)\n",
    "\n",
    "Recall the REINFORCE algorithm above. We compute the gradient of $Q = \\mathop{\\mathbb E} \\sum_t r(a_t, s_t)$ w.r.t. the parameter to update the policy. A natural question is that, when you take an inferior action that lead to positive expected return, the policy gradient is also positive and you will update your network toward this action. At the same time a potential better action is ignored.\n",
    "\n",
    "To tackle this problem, we introduce \"baseline\" when computing the policy gradient. The insight behind this is that we want to optimize the policy toward an action that are better than the average performance when taking all action into consideration at a given state.\n",
    "\n",
    "We introduce $b_{t} = \\mathbb E_{a_t} \\sum_{t'}{\\gamma^{t'-t} r(s_{t'}, a_{t'})}$ as the baseline. It average all possible actions at state $s_t$. So that the profit achieved by action $a_t$ can be evaluated via $\\sum_{t'=t} \\gamma^{t' -t}r(a_{t'}, s_{t'}) - b_t$\n",
    "\n",
    "Therefore, the policy gradient becomes:\n",
    "\n",
    "$$\\nabla_\\theta Q =\\cfrac{1}{N}\\sum_{i=1}^N [( \\sum_t  \\nabla_\\theta \\log \\pi_\\theta(a_{i,t}|s_{i,t}) (\\sum_{t'} \\gamma^{t'-t} r(s_{i,{t’}}, a_{i,t‘}) - b_{i, t})]$$\n",
    "\n",
    "In our implementation, we estimate the baseline via an extra network `self.baseline`, which has same structure of policy network but output only a scalar value. We use the output of this network to serve as the baseline, while this network is updated by fitting the true value of expected return of current state: $\\mathbb E_{a_t} \\sum_{t'}{\\gamma^{t'-t} r(s_{t'}, a_{t'})}$\n",
    "\n",
    "In implementation, we use a trick to match the distribution of baseline and values. During training, we first collect a batch of target values: $\\{t_i= \\mathbb E_{a_t} \\sum_{t'}{\\gamma^{t'-t} r(s_{t'}, a_{t'})}\\}_i$. Then we normalzie all targets to standard distribution with mean = 0 and std = 1. Then we ask the baseline network to fit such normalized targets.\n",
    "\n",
    "When computing the advantages, instead of using the output of baseline network as the baseline $b$, we firstly match the baseline distribution with action values. The transformed baselines $b' = f(b)$ should has the same means of action values and same STD of action values too. Then we compute the advantage of current action: $adv_{i,t} = \\sum_{t'} \\gamma^{t'-t} r(s_{i,{t'}}, a_{i,t'}) - b'_{i, t}$\n",
    "\n",
    "By doing this, we mitigate the instability of training baseline.\n",
    "\n",
    "Hint: We suggest to normalize an array via: `(x - x.mean()) / max(x.std(), 1e-6)`. The max term can avoid potential risk of error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "pg_with_baseline_default_config = merge_config(dict(), pg_default_config)\n",
    "\n",
    "class PolicyGradientWithBaselineTrainer(PGTrainer):\n",
    "    def __init__(self, config=None):\n",
    "        config = check_and_merge_config(config or {}, pg_with_baseline_default_config)\n",
    "        super().__init__(config)\n",
    "\n",
    "    def build_model(self):\n",
    "        # Build the actor in name of self.policy\n",
    "        super().build_model()\n",
    "        \n",
    "        # [TODO] Build the baseline network using Net class.\n",
    "        # Remember that you need to set the output dimension to 1.\n",
    "        # Remember using model.to(device) function to move your model to GPU (if applicable)\n",
    "        self.baseline = Net(self.obs_dim, self.act_dim, self.config[\"hidden_units\"]).to(self.device)\n",
    "        \n",
    "        self.baseline_loss = nn.MSELoss()\n",
    "            \n",
    "        self.baseline_optimizer = torch.optim.Adam(\n",
    "            self.baseline.parameters(),\n",
    "            lr=self.config[\"learning_rate\"]\n",
    "        )\n",
    "\n",
    "    def process_samples(self, samples):\n",
    "        # Call the original process_samples function to get advantages\n",
    "        tmp_samples, _ = super().process_samples(samples)\n",
    "        values = tmp_samples[\"advantages\"]\n",
    "        samples[\"values\"] = values  # We add q_values into samples\n",
    "\n",
    "        # [TODO] flatten the observations in all trajectories (still a numpy array)\n",
    "        obs = np.concatenate(tmp_samples[\"obs\"])\n",
    "        \n",
    "        assert obs.ndim == 2\n",
    "        assert obs.shape[1] == self.obs_dim\n",
    "        \n",
    "        obs = self.to_tensor(obs)\n",
    "        samples[\"flat_obs\"] = obs\n",
    "\n",
    "        # [TODO] Compute the baseline by feeding observation to baseline network\n",
    "        # Hint: baselines turns out to be a numpy array with the same shape of `values`,\n",
    "        #  that is: (batch size, )\n",
    "        baselines = self.to_array(self.baseline(obs).mean(axis=1))\n",
    "        \n",
    "        assert baselines.shape == values.shape\n",
    "        \n",
    "        # [TODO] Match the distribution of baselines to the values.\n",
    "        # Hint: We expect to see baselines.std almost equals to values.std, \n",
    "        #  and baselines.mean almost equals to values.mean\n",
    "        baselines = (baselines - baselines.mean()) / max(baselines.std(),1e-6)\n",
    "        baselines = values.mean() + max(values.std(),1e-6) * baselines\n",
    "\n",
    "        # Compute the advantage\n",
    "        advantages = values - baselines\n",
    "        samples[\"advantages\"] = advantages\n",
    "        process_info = {\"mean_baseline\": float(np.mean(baselines))}\n",
    "        return samples, process_info\n",
    "\n",
    "    def update_model(self, processed_samples):\n",
    "        update_info = {}\n",
    "        update_info.update(self.update_baseline(processed_samples))\n",
    "        update_info.update(self.update_policy(processed_samples))\n",
    "        return update_info\n",
    "\n",
    "    def update_baseline(self, processed_samples):\n",
    "        self.baseline.train()\n",
    "        obs = processed_samples[\"flat_obs\"]\n",
    "\n",
    "        # [TODO] Normalize the values to mean=0, std=1.\n",
    "        values = processed_samples[\"values\"]\n",
    "        values = (values - values.mean()) / max(values.std(),1e-6)\n",
    "        \n",
    "        values = self.to_tensor(values[:, np.newaxis])\n",
    "        \n",
    "        baselines = self.baseline(obs)\n",
    "\n",
    "        self.baseline_optimizer.zero_grad()\n",
    "        loss = self.baseline_loss(input=baselines, target=values)\n",
    "        loss.backward()\n",
    "        \n",
    "        # Clip the gradient\n",
    "        torch.nn.utils.clip_grad_norm_(\n",
    "            self.baseline.parameters(), self.config[\"clip_gradient\"]\n",
    "        )\n",
    "        \n",
    "        self.baseline_optimizer.step()\n",
    "        self.baseline.eval()\n",
    "        return dict(baseline_loss=loss.item())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== PolicyGradientWithBaselineTrainer CartPole-v0 Training Start ===\n",
      "Config:\n",
      "  checked: true\n",
      "  clip_gradient: 10.0\n",
      "  env_name: CartPole-v0\n",
      "  evaluate_interval: 10\n",
      "  evaluate_num_episodes: 50\n",
      "  gamma: 0.99\n",
      "  hidden_units: 64\n",
      "  learning_rate: 0.01\n",
      "  max_episode_length: 200\n",
      "  max_iteration: 1000\n",
      "  normalize_advantage: true\n",
      "  seed: 0\n",
      "  train_batch_size: 200\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([209, 1])) that is different to the input size (torch.Size([209, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== PolicyGradientWithBaselineTrainer CartPole-v0 Iteration 0 ===\n",
      "Training Progress:\n",
      "  baseline_loss: 1.0812811851501465\n",
      "  evaluate_reward: 22.0\n",
      "  iter_episodes: 14\n",
      "  iter_time: 0.6801052093505859\n",
      "  iter_timesteps: 209\n",
      "  iteration: 0\n",
      "  mean_advantage: 1.3689104960690202e-08\n",
      "  mean_baseline: 1.6426926663370978e-07\n",
      "  mean_log_prob: -0.6853688359260559\n",
      "  performance: 14.928571428571429\n",
      "  policy_loss: 7.562032699584961\n",
      "  total_episodes: 14\n",
      "  total_time: 0.6801052093505859\n",
      "  total_timesteps: 209\n",
      "  training_episode_length:\n",
      "    max: 29.0\n",
      "    mean: 14.928571428571429\n",
      "    min: 10.0\n",
      "  training_episode_reward:\n",
      "    max: 29.0\n",
      "    mean: 14.928571428571429\n",
      "    min: 10.0\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([212, 1])) that is different to the input size (torch.Size([212, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([201, 1])) that is different to the input size (torch.Size([201, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([221, 1])) that is different to the input size (torch.Size([221, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([226, 1])) that is different to the input size (torch.Size([226, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([215, 1])) that is different to the input size (torch.Size([215, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([258, 1])) that is different to the input size (torch.Size([258, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([274, 1])) that is different to the input size (torch.Size([274, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([203, 1])) that is different to the input size (torch.Size([203, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([300, 1])) that is different to the input size (torch.Size([300, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([234, 1])) that is different to the input size (torch.Size([234, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== PolicyGradientWithBaselineTrainer CartPole-v0 Iteration 10 ===\n",
      "Training Progress:\n",
      "  baseline_loss: 0.8948265314102173\n",
      "  evaluate_reward: 56.28\n",
      "  iter_episodes: 4\n",
      "  iter_time: 0.16256213188171387\n",
      "  iter_timesteps: 234\n",
      "  iteration: 10\n",
      "  mean_advantage: 8.151062935723985e-09\n",
      "  mean_baseline: -3.056648623100955e-08\n",
      "  mean_log_prob: -0.6192816495895386\n",
      "  performance: 58.5\n",
      "  policy_loss: -9.629011154174805\n",
      "  total_episodes: 80\n",
      "  total_time: 3.3439760208129883\n",
      "  total_timesteps: 2553\n",
      "  training_episode_length:\n",
      "    max: 80.0\n",
      "    mean: 58.5\n",
      "    min: 30.0\n",
      "  training_episode_reward:\n",
      "    max: 80.0\n",
      "    mean: 58.5\n",
      "    min: 30.0\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([217, 1])) that is different to the input size (torch.Size([217, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([228, 1])) that is different to the input size (torch.Size([228, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([248, 1])) that is different to the input size (torch.Size([248, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([247, 1])) that is different to the input size (torch.Size([247, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([293, 1])) that is different to the input size (torch.Size([293, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([265, 1])) that is different to the input size (torch.Size([265, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([232, 1])) that is different to the input size (torch.Size([232, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== PolicyGradientWithBaselineTrainer CartPole-v0 Iteration 20 ===\n",
      "Training Progress:\n",
      "  baseline_loss: 0.7612990140914917\n",
      "  evaluate_reward: 100.64\n",
      "  iter_episodes: 3\n",
      "  iter_time: 0.1655571460723877\n",
      "  iter_timesteps: 232\n",
      "  iteration: 20\n",
      "  mean_advantage: 8.22133028322014e-09\n",
      "  mean_baseline: -3.0829989672298552e-09\n",
      "  mean_log_prob: -0.5811368823051453\n",
      "  performance: 77.33333333333333\n",
      "  policy_loss: -12.29688835144043\n",
      "  total_episodes: 117\n",
      "  total_time: 7.144812822341919\n",
      "  total_timesteps: 4948\n",
      "  training_episode_length:\n",
      "    max: 96.0\n",
      "    mean: 77.33333333333333\n",
      "    min: 62.0\n",
      "  training_episode_reward:\n",
      "    max: 96.0\n",
      "    mean: 77.33333333333333\n",
      "    min: 62.0\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([205, 1])) that is different to the input size (torch.Size([205, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([244, 1])) that is different to the input size (torch.Size([244, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([208, 1])) that is different to the input size (torch.Size([208, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([240, 1])) that is different to the input size (torch.Size([240, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([213, 1])) that is different to the input size (torch.Size([213, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([252, 1])) that is different to the input size (torch.Size([252, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([269, 1])) that is different to the input size (torch.Size([269, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([202, 1])) that is different to the input size (torch.Size([202, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([294, 1])) that is different to the input size (torch.Size([294, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== PolicyGradientWithBaselineTrainer CartPole-v0 Iteration 30 ===\n",
      "Training Progress:\n",
      "  baseline_loss: 0.33726486563682556\n",
      "  evaluate_reward: 105.32\n",
      "  iter_episodes: 2\n",
      "  iter_time: 0.24734091758728027\n",
      "  iter_timesteps: 294\n",
      "  iteration: 30\n",
      "  mean_advantage: -1.1758738871492369e-08\n",
      "  mean_baseline: -9.082612706379223e-08\n",
      "  mean_log_prob: -0.5378826260566711\n",
      "  performance: 147.0\n",
      "  policy_loss: -7.346587181091309\n",
      "  total_episodes: 140\n",
      "  total_time: 12.334936141967773\n",
      "  total_timesteps: 7284\n",
      "  training_episode_length:\n",
      "    max: 200.0\n",
      "    mean: 147.0\n",
      "    min: 94.0\n",
      "  training_episode_reward:\n",
      "    max: 200.0\n",
      "    mean: 147.0\n",
      "    min: 94.0\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([262, 1])) that is different to the input size (torch.Size([262, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([296, 1])) that is different to the input size (torch.Size([296, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([358, 1])) that is different to the input size (torch.Size([358, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([295, 1])) that is different to the input size (torch.Size([295, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([237, 1])) that is different to the input size (torch.Size([237, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([306, 1])) that is different to the input size (torch.Size([306, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([382, 1])) that is different to the input size (torch.Size([382, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([312, 1])) that is different to the input size (torch.Size([312, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== PolicyGradientWithBaselineTrainer CartPole-v0 Iteration 40 ===\n",
      "Training Progress:\n",
      "  baseline_loss: 0.13112188875675201\n",
      "  evaluate_reward: 138.08\n",
      "  iter_episodes: 3\n",
      "  iter_time: 0.21841692924499512\n",
      "  iter_timesteps: 312\n",
      "  iteration: 40\n",
      "  mean_advantage: -9.934107758624577e-09\n",
      "  mean_baseline: 3.667978276666872e-08\n",
      "  mean_log_prob: -0.5285580158233643\n",
      "  performance: 104.0\n",
      "  policy_loss: -6.389889717102051\n",
      "  total_episodes: 167\n",
      "  total_time: 17.954907178878784\n",
      "  total_timesteps: 10154\n",
      "  training_episode_length:\n",
      "    max: 115.0\n",
      "    mean: 104.0\n",
      "    min: 82.0\n",
      "  training_episode_reward:\n",
      "    max: 115.0\n",
      "    mean: 104.0\n",
      "    min: 82.0\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([235, 1])) that is different to the input size (torch.Size([235, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([299, 1])) that is different to the input size (torch.Size([299, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([344, 1])) that is different to the input size (torch.Size([344, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([301, 1])) that is different to the input size (torch.Size([301, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([348, 1])) that is different to the input size (torch.Size([348, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([288, 1])) that is different to the input size (torch.Size([288, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([400, 1])) that is different to the input size (torch.Size([400, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\loss.py:431: UserWarning: Using a target size (torch.Size([384, 1])) that is different to the input size (torch.Size([384, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== PolicyGradientWithBaselineTrainer CartPole-v0 Iteration 50 ===\n",
      "Training Progress:\n",
      "  baseline_loss: 1.120503544807434\n",
      "  evaluate_reward: 196.42\n",
      "  iter_episodes: 2\n",
      "  iter_time: 0.2732675075531006\n",
      "  iter_timesteps: 400\n",
      "  iteration: 50\n",
      "  mean_advantage: 1.2278556482669956e-07\n",
      "  mean_baseline: -2.527236802052357e-07\n",
      "  mean_log_prob: -0.5275930762290955\n",
      "  performance: 200.0\n",
      "  policy_loss: -12.943634033203125\n",
      "  total_episodes: 187\n",
      "  total_time: 24.94820737838745\n",
      "  total_timesteps: 13553\n",
      "  training_episode_length:\n",
      "    max: 200.0\n",
      "    mean: 200.0\n",
      "    min: 200.0\n",
      "  training_episode_reward:\n",
      "    max: 200.0\n",
      "    mean: 200.0\n",
      "    min: 200.0\n",
      "\n",
      "In 50 iteration, current mean episode reward 196.420 is greater than reward threshold 195.0. Congratulation! Now we exit the training process.\n"
     ]
    }
   ],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "pg_trainer_wb, pgwb_result_wb = run(PolicyGradientWithBaselineTrainer, dict(\n",
    "    learning_rate=0.01,\n",
    "    max_episode_length=200,\n",
    "    train_batch_size=200,\n",
    "), 195.0)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "g:\\Anaconda3\\envs\\torch\\lib\\site-packages\\ipykernel_launcher.py:6: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version\n",
      "of pandas will change to not sort by default.\n",
      "\n",
      "To accept the future behavior, pass 'sort=False'.\n",
      "\n",
      "To retain the current behavior and silence the warning, pass 'sort=True'.\n",
      "\n",
      "  \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Policy Gradient')"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEXCAYAAACH/8KRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsnXecJFd97b+3c/d0Tw47m3e1u5JWWQgkAQIJCQMySORgkgkGjPww5vGMjW3AYByewWCwCSI8wAKBhCyBSUICSUiIlVBYxdVGaXd2ZnbyTIfp3Pf9cetWVXdXh4k7u1Pn89nPzFRXVVeHvafO+SUhpcSFCxcuXLiYLzzH+wJcuHDhwsWJDZdIXLhw4cLFguASiQsXLly4WBBcInHhwoULFwuCSyQuXLhw4WJBcInEhQsXLlwsCC6RuDipIYT4hBDiOuP3jUKIpBDCe7yva64QQlwqhDhq+/sJIcSlx/GSXLgw4RKJixMCQohnhBBpgwhGhBD/TwgRncs5pJRHpJRRKWVxka8tJoT4N+MaU0KII0KIHwohnrOYz2OHlPIMKeWdCz2PnWhduJgvXCJxcSLhFVLKKHA+8Gzgb4/z9SCECAK/Bs4CXg60AqcD3weurHGMb9ku0IWLZYBLJC5OOEgpB4GfA2cCCCHWCiF+LISYFEIcEEL8idNxQojNQgipF3IhRKehbIaEEFNCiFuM7Y8LIV5hO84vhBgXQpzrcNq3AuuBV0opH5dSFqWUKSnlD6WUn7CdQwohrhFC7Af2G9v+XQgxIISICyEeFEJcYts/LIT4lnFdT6KI0/5anhFCXGH87hFC/JUQ4qAQYkIIcYMQorPiNb/dUErjQoi/MR57KfBR4A2G0ntkLp+DCxcaLpG4OOEghNiAutt/2Nh0PXAUWAu8FvhHIcTlTZzqv4AIcAbQC3zO2P4d4C22/a4EhqWUux3OcQVwq5Qy1cTzvRK4ENhp/P174FygE/gecKMQImQ89nHgFOPfS4C31znvB4xzvxD1HkwB/1mxz/OBU4HLgY8JIU6XUv4C+EfgB4bld04Tr8GFiyq4ROLiRMItQohp4B7gLhRhbEAtkh+RUmaMxf7rKKVQE0KIfuBlwPuklFNSyryU8i7j4euAK4UQrcbfb0WRjhO6gWO2854rhJg2VMbein3/SUo5KaVMA0gpr5NSTkgpC1LKzwJB1GIP8Hrg08b+A8AX6ryc9wJ/I6U8KqXMAp8AXlthof29lDItpXwEeARwScPFosElEhcnEl4ppWyXUm6SUr7fWJDXApNSyoRtv8PAugbn2mAcN1X5gJRyCPgt8BohRDuKcL5b4zwTQL/t2N1Synbg1ShisGPA/ocQ4n8LIfYIIWYMgmxDERPG67Lvf7jOa9kE3GwQ2DSwBygCfbZ9jtl+nwXmlKjgwkU9uETi4kTHENAphIjZtm0EBhscN2Ac117j8W+j7K3XAb8z4jJO+BXwB0KIliau1Wy1bcRDPoJSHh0G+cwAwthlGEV2GhvrnHcAeJlBsvpfqM41O16TCxfzhUskLk5oGLbPvcA/CSFCQoizgXdRW0Ho44ZRAfsvCSE6jID6C2y73ILKDvtzVMykFr6DWvRvFkKcKYTwGnGOCxpcegwoAGOATwjxMVTGl8YNwF8b17Ye+F91zvUV4NNCiE0AQogeIcTVDZ5fYwTYLIRw1wIX84b75XFxMuBNwGaUOrkZ+LiU8rYmjnsrkAeeAkaBD+oHDNvsJmAL8N+1TiClzACXAU8CPwXiwF5UltXr6zz3rSgi24eyrTKUW1l/b2x/GvgltWM0AP8O/Bj4pRAiAexCBfWbwY3GzwkhxENNHuPCRRmEO9jKhQtnGCphh5TyLQ13duFiFcMtjHLhwgFGHca7aJD95cKFC9facuGiCkZB4wDwcynlb4739bhwsdLhWlsuXLhw4WJBcBWJCxcuXLhYEE7oGEl3d7fcvHnz8b4MFy5cuDih8OCDD45LKXsW63wnNJFs3ryZBx544HhfhgsXLlycUBBC1OuUMGe41pYLFy5cuFgQXCJx4cKFCxcLgkskLly4cOFiQXCJxIULFy5cLAgukbhw4cKFiwVhyYhECLFBCHGHMW/hCSHEnxvbO4UQtwkh9hs/O4ztQgjxBWNU6qNCiPOX6tpcuHDhwsXiYSkVSQH431LK04GLgGuEEDuBvwJ+JaXcjprl8FfG/i8Dthv/3gN8eQmvzYULFy5cLBKWrI7EmPcwbPyeEELsQU2tuxq41Njt28CdqAE/VwPfkapnyy4hRLsQot84j4vjhfQ0PHEzlArWts2XQO9p1fvuvx2mnrb+DnfAma8BIcr3KxXhsRshm6ApBGNw1uvA4yWVT/HI2CM8d+1zq3bLl/L85OBPyBaz5raIL8yVhQA/Sp7JbL5obveU8vRN/p7h7urztEUCvOLsfkTFdd+zf5wLNncQ8nsdLzOdK7Lr6Qku3dFTdWypJLltzwiXndpLwFd9/3bP/nHO2dBGLOQv214olrhl9xDpXKHqmHroH/8dsdkjZds8QnDGulYifi9EuuDMVwPw6NFpHhmYrnmurT1Rnretu2r7E0MzPHS4asCkifUdES47rbdqe7ZQ5J7947zotN6y92lwOs30bI4z1rY1fH2rHTc/fJT+tjAXbe063pcCLFNBohBiM3AecB/Qp8lBSjkshNDftHWUz2M4amwrIxIhxHtQioWNG+sNjXOxKLjrX2DXl8q3bXkhvP3H5dvyafje60EWy7eH2mH7FeXbHvsh3PzeuV1H+ybYdDE/PfRTPrXrU1x35XWc01M+dvzm/TfzqV2fKtvWH+zkqqd2893sJ3hI7jC3/6FnF/8Z+ALPz/47Rx0KfHf2x9jWaw1dHJ5J85Zv3Me/vvZsXnfBhqr9AW594hgf/MFuvvTm87nyrP6yx77/+wE+evNjfOqVZ/LWizaVPbZnOM5bvnEff/fynbzr+VvKHvvtwQk+fOMjdd6Yavgo8GTwGgKiWP3gk8bPNWeZRPKXP3yUp47VJvWgz8OeT74Uj6ecHD9y06M8Phivey0/+8Al7FzbWrbtP359gC/++gC3f+iFbOu1Jv7+2y/3cc+BMe776BWVp3FhQ6kk+Yef7OEFO3pWD5EIIaKoAUEflFLGK+/U7Ls6bKvqKCmlvBa4FuCCCy5wO04uJbJJePg62Hk1XPlZte0nH4Sxp6r3nTmqSOTKz8DOV6rfv/oCRUJ2IpES7v0idJ8Kf/wTnD92GyYOwP97KcTV1NjJzCQAN+y9oYxIpJRc/9T1nN55Ol++wnJFPXt/Dk9dQ1Sk+d67LmTHGkUOod1H4Xb4n/ecQ7Fnp7n/vQcn+MD1DxPPlCuAyVROXY7x0wkz6TwA//KLp7ji9D5TeSQyef7ttr0A/M8jQ1VEcsvD6rUNTM5WnfPwRAqAWz/4ArqigZrPbYdn+hkCXy+SfNGnyZ76SkD9R3rZ53/DO563hfdfug08lqqaTOV45blr+duX76w61w9+P8C/3rqX6XSezpby5x+YTPO6Z63nIy+rVqfpXJErv3A3n799H9e+zRoWOZ7M8o17njb3sSOeyTMSzzKayNAbCzX1WlcjHhucYSKV44U7Fq3DyYKxpEQihPCjSOS7Uko9ZW5EW1ZCiH7UZDpQCsR+q7ceNfHOxfHCo9+HbBwu/jOIGl/a9k1w8A5FCPabgmnDRundae377D+BO/4BRp+yrLBDd8DIY3DVf0C02vaogtewehLHAEjmkgDc+syt/OWz/5K2oLJB7j92PwemD/Cp532KrrDtLq2oCCFAgf72MN3RoNoulP3VEQT0NqAvpn6vXOQSBrHEDbJwwqxxzOGJWa7bdZh3GuriP+84yHgyx4t39nH7nhGOzWRY06YWymJJ8qPd6ms+NJ2uOueRiVmCPg87+qJVdllNjCoRH91wNtG+9db1+TuZoM36fAwkswW6o0HrvbFhc5caRT+ayJQRSSpbYCadZ0tPi+NxAO9+/lY+d/s+Hjs6w1nr24z34oD5PuWKpbL9M4b1+ORQnN5Ty4kklS1QkrLK+luNuHPvGELAC1YQkSxl1pYAvgHskVL+m+2hHwNvN35/O/Aj2/a3GdlbFwEzbnzkOEJKuO9a6D8X1j/b2h5bA/lUdXxj5qj62W67F7jgHeANwn1fsbbd+0WI9sHZ9abQ2hBqA18YEuqrkMgn8Hl8ZItZbjlwi7nb9/Z8j45gBy/b8rLy49PK+w+SJxKwxTbyxt1/sZwYwsY+lUSSNIhkpi6RFBACnretiy/8ej8z6TxHJmb55j1P85rz1/ORl56GlPDTx6yv9X2HJjgWzxDweRieyVSdc2Bqlg2dkeZJBCxSby+3fsMBr7mIaxSKJWZzxZoLdG+rIomReLZs+/CMIr117eGal/GO52+mLezn87fvA1QM5Lu7jrClu8V8bjv0e/7EULVd9qEbdvP+77qTgAHu3DfK2evbqxTi8cRSZm09DzVd7kVCiN3GvyuBfwZeLITYD7zY+BvgZ8Ah4ADwNeD9S3htLhrh0J0wvhcufG+58ogZ3r+hEEzMDIDwQGytta2lWxHGI9+H2Uk49hgc/LU6p8/5LrYKQkCsz3y+RC5Bb2gdGyI7uXHfjZRkicHkIHcevZPX7HgNQW/FeTMzAATImyQBgKFsKJZbVZps0vkKRZJVBFJpedkxmysS8Xv56JWnM5PO86U7D/DPv9iD1yP4y5eeyrbeKDv7W/nJo5bQvvnhQaJBH394Vr+5ONtxZDLNxs5InTfIAVOHQXihdX3Z5nDAa971aySz6vVEQ87mRK+h0Ebj5SQ3OK3+7m+rTSStIT/vecFWfvXUKLsHpvl3g1D+/PLtAOSL5c60fs+fHC4nknyxxN37xxmJVxPtasNUKsfugWkuXUFqBJY2a+seahvglzvsL4Frlup6XMwR91+rMnvOeHX59tga9TMxDD1W8JrpAUUi3oqv1EV/Cg//Fzz4LRjbC/4WuOCdc7uWWD8kR9TT5hLMZvyMDZ2Nf833uf/Y/dw7eC8CwRtOfUP1sZpIRIGwPdsqpxVJOZHojKz5WlvhgI8z1rbxqvPW8Y27n6ZQknzoxTvoa1VWzcvP6ef//mIvA5Oz9MSC/PzxY7z0zDVs7Ixw88ODZAtFgj51DVJKjk7OcuGWzibfKAPTR6B1XdVnEfZ7a76uWE0iUdc9mqhQJIYNt7a9fizj7c/dzNfvPsTf3PwYe4bjvP25m9nUpYgxX6pQJDZry47HBmeYzRWryH014u4D40gJLzx1ZRGJW9nuohpTz8Den8Oz/hj8FQtFq6E4EhWu48zRcltLo+8M2HqpCro//kM4/20qLXguiK0xny+ZS+KRETLTZxDzt/GdJ77DTftv4kUbX8SaljXVxxpEEvYU8HttX/ecCmJXWluRgFpQqxRJE9ZWOlcwFc2H/+BUvB5Bf1uIP7lkq7nPK85W799PHxvm9j0jJLMFXnXeOvqNmMkxm701PZsnkS2wvqP2Xb8jpo9Ax6aqzWG/t+braq1BJOGAl1jIx1gFkQxNp/EITIKshWjQx3tfeApPDMUJ+b1cc9k283PIFypiJAbJPTORMpUSwK5DEwCkc+X7r0bcuXeUjoifc9a3H+9LKYNLJKsFt30M7vznxvsB/P7ryqa64F3Vj0X71M8qIjkCbc5psVz0fkiNqbjLRX/a/DVrxPotayufQMgQSD/nd76EuwfvJp6L80ebroTBB6uPNYikxVtxN5vXRFKuSLRqqYwlxDP5sp9OmM0VTSJZ2x7mO+98Dt9+53PKLLUNnRHO3dDO/zwyxC0PD9HXGuSirV1mrGHQFnA/YmRxzdnamj5cFR8BpbaqFYl6PdFg7SB2byxYZSsNzajMqjJyroG3XbyJTV0RrrlsG93RoEUkDtbWuvYwUsJTNntr1yGVqVdpy602lEqS3+wb45LtPXg9c4iZLQNcIlkt2PdLePruxvsVcvDQd+D0V0DbuurHg1EItpbHSEpFiA9B2/rq/QG2vRj6zoRz3uh4p9wQ0T4V08gmSOQSiJJadNeIFwKwo2MHz3ryF3Dda6uPrUUkOWciCfnVf4nKO/dk09aWRRoXbu1iR1+sar+Xn93PE0Nx7tg7ytXnrlPKxSCS4WlrwR6YUkSyYS5EUsgqkncgkkigWpHoO/9a1hYoe6vS2hqaTje0tazn9XHX/7mMay7bBoDPqxbBgoO19axNSq3qgHu+WOLBZybNx5UDvjrxxFCc8WSOS1eYrQUukawepCehmG28X2pULb6nXFZ7n9gaRRwaiWOq8t3J2gLweOA9d8JVX5zLFdueTwX4ZfwYiVwCWVIL2MhkjI9e+FH+9qK/RYzvU6+xYnEio7K2WjyVROKctSWEMGIJ5UF1K0ZSqLmYzdqsrXp4+dlrEUKl/r7yXEXW2tqyB9y1IpkTkZjZcw7WlgORNIqRAPS1BhlNVCiS6TRr62Rs1UPAUCQ5m7VVKkky+RKbu1vobAmYcZLHB2dI5Yrs7G+lWJJVKmY14c69qlLiku0ukbg4HpBSZU0VmiCSWXX3R7hOgNdmNQEqYwtqW1ug6kE8jRdZ5+dTsY/szBHypTylolrA9o8keNNpb+K83vNg0mjNorOxNHSMxFuRbaUVicN74nTnri2gXLFEJu/s1Strq3H+ypq2EM89pYud/a2c3m8USPq9dLYEGLLFSAYm03S1BIgG55ATM/WM+jlXa6ueImkNMRrPmgQqpWRoJjNvItHWVqFkkULWIJVIwMvO/laeGFafm7a19F34ag6437VvjLPWtdETazLjcRnhEslqQC4JpXzV3bcj0kbvpHoB8SoiMe6C6xHJQmAokuSMqo8oFdR/pENjKVWLkM9YZFaDSCKeCiKpESMBteBWxkgStrTfWnGSdL7YlCIB+PJbnsV3331hWX1If1uorChxYHKW9XOOjzjXkEANgszqYHv9GEm2UCKeVvtOpHLkCiXWts2v+lxbW3lbHYm+rpDPwxlrW9l3LEm+WGLXoQm290ZZ36Heh9UaJ5mZzfPQkakVaWuBSySrA1plNGNtaSKJ1FMkRhaVtnj04lUrRrJQGIokHldkUSgE8XkEuWKJZyZmVXBZd9OxF0rmM1BQd/jhKmvLOWsLnOst7ERSK3PLHmxvhNaQn46KgrK17eGqGMncA+1HwOOzsutsqJX+6/cKgg6NJDV6W3UKsLo2fY0LVSR2m0oTSTjgZefaVnLFEnuPJXjgmUku3NpJOGDEriqu/9dPjXD3/rF5XceJhLsPjFGSuETi4jgibRBJoXafqKp9GymSUh5mVVomMwNq/2C09jELQTAG/giJpMoUK+SDnGr0zNo/koCJg9a+diLJWpk/IVFpbTnXkYBx515Z2Z4tmMV5tQLus9kCYf/8S7PWtoUYMmIkhWKJwak0G+ac+ntYEbqDjajTf0s2SymRyRMN+upWzptFiUbAfdCsIZkvkVQrEk3cIb+XM4wmjzc8MEAqV+SirV1mNl2lovrsL/fxkR8+WvaaTkbc/uQI7Ssw7VfDJZLVgPkoknpE0qqr240U4JmjS2drgVHdvoZkSgUbc/kgZ6xtRQjYN5KESTuR2IrZ0lZr9DIikdJW2V5NCk7WVjyTZ52xqDtZW1JKZudgbTmhvz1MIlMgmS0wPJOhUJLzUyQOthZA2IjfZG1B7mSm0LB/lSYSnQI8tGAiMWIkdmvLeL/Dfi9buqOE/V5ufEBZphdu6bIKRSuIZCadZ2gmw/1GZtfJiFyhxK/2jPLi0/vwNZFufTywMq/KxeJCk0MzimR2UvW28tdZJCrbpEwP1Fy8Fg2xfhJppYByuQAdkQAbOiLsG03A5CFrv6wtRmLERwACwrb4FzKYVlgNRWK3tkolSTJbMGs9nKytbKGElBAJLoBIdObWdHp+qb+g2qM4ZGwBhB1SmxOZQt2MLbBbW+pGZHgmTdDnoSMyvwaKPqMGImeztjI2a8vrEZzWHyOdL7KtN0pPLGgqkkwlwRufhe6gfCLgwGiCxwdnGu9o4N6D4ySyBV56pkPB7QqBSySrAZpImlIk040rz3WblPiQurufGVi6+IjtORMZ9TqyuSBBv5cdfVHL2moxvGO7tWUjkqBdkej4CDgSSbhCkaRyBaTEDPjqoLMdev9IjaFXzcBelDgwn2LEfFqlb9cikoAutrSuvxkiiQZ9tAS8jBqNG4emM6xrD8+tkaQNQgj8XuEYbNeEsbNf2VsXbe0su/Z0BcHrZIGfPjZ8QgTipZS869sP8PIv3sMffW0X9x4Yb1gbc+sTx2gJeB2Hi60UuESyGqCtrULWCpDXQnqyfqAdIKr7bR1TdRq55NJaW8ZzJnKKJGQxRNjvZUdfjENjKeTkIeg3ZpOUEYmytuIyTIBaROIcbK+8awdMa8tJkejFuZn031owixJnMgxMps0WK01j2shca2Bt2RfceCZft6pdo7c1ZAbbh2bmX0Oi4fN4HK0tbWHpKYl6cJNTjCRpEPxlp/aQyBT49VOjrHQcGk9xeGKWy0/rZf9okj/6+n284au7GE863+QVS5JfPjHCi07vqzmZcyXAJZLVAB1AR6oq9Lr7TjVWJL4ARLpVjEQvXsuhSCjgFR6QAUJ+Dzv6YnhLWRWj0USSq1YkY7KdALbFvwlFYg+26+rvjoifSMDrGGw3Pf4FxEj6YkE8QllbRyZnWdsempsnPn1Y/azRPcBcjG09q5LZQs0+W3b0xII2RZKeG8E5QCkS56wtgD84o48/unAjl52qZtY4NdPUn8NLzlhDbyzIzSeAvXXnXpVh9omrzuDuv7yMT119Bo8OTvPGa3dVdVgGeOCZSSZSOV56xsq1tcAlktWBWVsgspG91QyRgFFLMuw8h2QpEOsn4fEQ8UYAQcjvZXtflPViDIGEntPAF3JUJOO04Ze2xT9vm0RYi0jKFInuR+WjLex3VCQpbW0tgEh8Xg+9sRBDM5l5pv4aRFJLkfjnZ22BCriPJjLkCiVGE9kFK5KAz+OYtaWvsTsa5B9fdRYtRjGmJhi7mtKfQ3skwNXnruXOvaNM1ZlguRJw595RtvVG2dAZIeT38taLN/OtdzyHoek0b7h2V9UogV88cYyAz7Ni0341XCJZDUjbiKRRdfvsZJNEYtSSmFXtSx1sX0PC4yHqURlEIb+HU3qibPUYAf/OU1SacEWMRHqDJGQYn7QtMPaiRQdrSxfuae86brYR8dMa8jtmbS2GtQXQ366KEgcmZ9nQMY+MLW/Ash4rUBlnkFIlEdSratfoa1X9tkbiGaSsP9CqGfg85URiz9pyvHYHa0vHqlrDPl513nryRVk2NGylYTZX4L5Dk1xWQQoXbe3iO+98DmOJLK//6u/M8cpSSm59/Bgv2N5jEupKhUskqwFliqTOHZuUSpE0ipGASgFOHFNE4gupIVZLiVg/SY+HFmOETsjnJeT3cl6L8do6t0AgWpW1VQy0ksOPz65IcnZFUk2soYAXKa00WXur9VqKJL0IigRUSu3BsSTjydz8MrbaNqjeZg4wM5+MxTidL1IsNTe+tjcWZDZXZP+oIur+Jhs21oLfJyiUWVvqva5lDYYcbDlN6K0hP6f3xzi1L7ai7a17D0yQK5a49NTqEdMXbO7kundfSCJT4Or//C33HhjnscEZhmYyKzpbS8MlktWAZhWJbqXSrLWVHFU9rtrWl09RXArE+kh4BC3G2qMXlp3BceLEINJJzhdleGzUtKLIzFAIxMjhw1eyKxIjRuLxOaf/Vvjx+nyxkJ/WsK9+1tZCiaQtZI61XcwaEqhWJM00bNTQI3d3H1F24UKtLb/HUzazXV9TrQp7r0cQ8HkqFIn6XNrCfoQQXH56Lw8dmVqxxYl37hslEvBywWbn/1/nbmjnR9c8j55okLd+837+7kdP4PUIrji9mnhWGlwiWSko5OBbL4cjuxb/3LNTVhPGeoqkmWJEjdgaQMLgQ0ufsQUQjJHw+ogai48mki2eYxwq9XL1f/6Wh0YKHB4a4Wfa3sjMkPe3kpUBPGUxEoNIwh01s7YAZo1FK2lbcFtD9RXJQoLtUD66ds6KZPpw3Tb9ETP9t5wgm2kKqSclPjxgEEmdEbvNwO/1lCmSTL5I2O+tm1Ic9nsrMs7K+4R1tgSQUqVrrzRIKblz7xjP29ZtTsB0wqauFv77/c/l0h09PDIwzcVbu2iPrJzZ7LWwZEQihPimEGJUCPG4bdsPbPPbnxFC7Da2bxZCpG2PfWWprmvFYnYCnrkbjvxucc9bLEB2xioirKdImun8q2EWJdaZQ7LISHh9RAtq8dNzQ/oKQzwt1yClpKeriyhpqy9WZoasTykSr5MiCXc4B9v1lMScdefuEWohbg07x0hSixQjsc/4mJMiySbVd6iOIqnMfEpkGjds1OgzFMkjA9N0RPwLJkxfZR1Jrmh+prVQmU2nFYmO8WhCtE9XXCk4OJbk6FS6qaB5LOTn2rddwKdfdSYfvfL0Zbi6hWMpIzjfAv4D+I7eIKU0h2oLIT4L2Ms7D0opz13C61nZMJoL2ovoFgVaZcTWwOgT9bO25qRI+q3fl7qq3UBCCFrz6vpDfi/kMwRTQ7z8hW/mVVc8n+KNazgysd9qAZKeJhPuI4fEYycMHSMJtTsrEgdrS/ejag37SWYLlEoSj21K3WJZW1qRtAS8c6sc10kPNYoRoTpGMhdrq8dQJPFMweyFtRD4vR7ypfL031qBdo3K+p54Jk8s6DOnBWpCSWYK0LbgS1xU6LRfp/iIE7wewZsvnMcQuOOEJVMkUsrfAI4NcITSr68Hrl+q5z/hoJVCJl5/v7lCx0f0wl+vlXwznX817ESyDIqkUCowKyRtBgmE/F6YPoxA4u/dDoAn1EpMpC37IzND2hslhw9Rqsja8gYgEKmZ/gvlsQQdkG4L+5GyvBswKNLxiNoef7PQsYcNnZG5VY5P6dTf2ouP3yvweoTN2lKvoZmsrdaQz3xt/Qu0tfS12Ge2p/NFQg1IOFgVIynQGrbIViuSxApUJHfsHWVHX3TB2W4rFccrRnIJMCKl3G/btkUI8bAQ4i4hxCW1DhRCvEcI8YAQ4oGxsZOoffRSKRJtV+m2JvWsrWY6/2q0dIMw/uMvQYxk1/AufvH0L8y/U0ZeyeEBAAAgAElEQVRcoz2XBKSyQXTX386tAIhQjCgGkUgJmRlmPVFy+BHFnFXVn58Ff0SRiaO1VU4kcVuthS7eq7S39FCr+bYN0ehqCRDweuYXaIe66lAIQcRWI5PMWkkEjSCEoM/oubVugRlbYCgSm7WVbVKRVNaR2IkkZlckKwipbIHfPz1lFleejDheRPImytXIMLBRSnke8CHge0IIR/0spbxWSnmBlPKCnp6VXaQzJ5iKZAmtLVi8YLvHq2apw5IUI379sa/zxYet0by6PUpbMUcrs2rRmSwnEgIxQiJPLpdTfadKeVKihaw07rj1e5xLqVRhb6CBtaUWpGQ2b8YR9MJVGXBP5wsLjhsAeDyCNz5nA394dn/jne2YPqzSsKP1F6uQbTGei7UFVhfghWZswTytrcoYSSZfVpWvCbFSLR5vPDkcJ1csme1eTkYsO5EIIXzAq4Ef6G1SyqyUcsL4/UHgILBjua/tuGLJYiQV1lbdYPsU+FvA1+Qoz9Z+QECseohSszgcP8y9g/dWbR9MDDJlNGkEi0hipRI9YlpZW5OHFOlpKy6oZpTITMJ8H5MiQl4Yd61FO5FE1PjfGt1/odza0vZPm0EklW1SUtmFtZC345NXn8nVxiz3pnHkd6q6v4Eisjek1FlPLU0mCOgU4P5FIZIKaytXbEjElR0H4hWKxAq2l3829+wf58BoxeTMZcShMfXcp/Qs0byeFYDjoUiuAJ6SUh7VG4QQPUIon0QIsRXYDhyqcfzJiaVSJJXWVqNgezNqRCPWrybx+eafnnjto9fy4bs+XLatWCpyLHWMRD5B3lAMdiLZIEaVXz9x0FIjYBKJyCdsDRtbwGsQo26jn0tBoKWhtWWPJVjWlrMiaXZe+5Jg8hAMPghnvKrhrvahXclMgagtWN0IOgV4saytQskeIyk1bEoYqrC2EplCWcaZJvtKRfKhG3bzudv2Lfia54tDYykCPo/Z9PNkxJJ984UQ1wOXAt1CiKPAx6WU3wDeSHWQ/QXAJ4UQBaAIvE9KefJOqnHCUioSj88iiHozSdKTEJkDkVz615BaWJxqIDFAIp8gnovTGlBu5lh6jIJUi8FUdoreSC+JvCKSoAzwFf/nEb9Mw/g+2GwLpxkTGkXWUiQzsgXpC6pvlSbR/KxSXl5/3ToSe9aWJpI2I5OqMkaSzhcWTZHMGY/fpH6e+ZqGu4Zsd/X219UMTEWyCMF2n9dT1rQx06S1lcnbKtvTeVrD1vVrZWVP/y2VJOPJLAfHjp8iOTiWYnNXpGnCPhGxZEQipXxTje1/7LDtJuCmpbqWEwJakWQXOWtrdlLVhWi7ajEVyZozF3ZtKAsLYDg5TGunIpKjCVOsMpUxiMRQJD/a8n8556nvc/WuL4EsOSoSTz5lEslUKYLwBhSRmDGSpOpH5Q04Wn329F8pZVnWlvbknRRJM4V9S4LHboINFzUVq7LHGZpt2Kjx6vPWE/Z7F9z5F6ieR5JrMkZikGDRmEViVyRej6Al4C0Ltk+n85Skat9eLMnjspgfGk+yoze27M+7nHAr21cKtCLJzzY3ybBZ6Pki2t5plP7bTDHiIiFTyDCaVjMkhlNWs72h1JD5+2RGCdOk0Whx0ruJfwx8AN6/Cy7+MzjnDdYJg4qIvPmEjUjCNhLV1tasESMJOr4ffq8Hn0eQzhfJFkoUStJccKNBHx5RPdyqmYVwSTDyBIztgbNe29TukYA9a6swJ/Jb0xbiHc/bsuDMNFAtUioHWzWMkVTYckBZjASUvWVXJJMpdaOQK5QYnCrvrHtoLMkNDwzwDz95krd/834+9qPHefDwZMNBU3NBvljiyMQsW3taFu2cKxEru6XkakLBNosgGwffIjVB1O1RdByjUWX7XBTJAmEnjKGk9ftg0mq8pwPuWpEU8kHC/hz0nAov+XT5CQPK2vLZFMlkMUzUr2Mk9mB7S81gO6hFazZXNC2smLHg6qJE5/Tf40Akj/0QhAd2Xt3U7qFAubV1vNpvVDdtLDaOkfitrsxWw8byJSwa9JXVkUwkrc/34FiSjV0qrTqTL/KHX7iHdL5I0OdhS3cLuw5N8J3fHWZde5g/v2I7r7+gtsLL5IsIQd12JwBHp9IUSpKtJ3GgHVwiWTmwL2iZmcXrppueVPaPt4G1NZfOv4sEbWsBHEsdK9se9oVJF9JMZRWRxHNxIr4IuQK1FxzD2vIXUpBRd6rjhRBb/IYVo9/jfMqIkTgH28Hq65SwtZDXcOq3NZsrEFlua0tKFR/Z8sKGab8aldbWnPt5LRJ8tqaNxZIkVyg1ZW2B6sqs3/9qReIvs7YmUuVEctlp6n16fHCGdL7Iv772bF59/nq8HkEyW+CXTxzjul2H+csfPsrA5CwfevGOKgX200eH+ejNj/HinX185nXn1L1mnbF1sisS19paKbArEiPjaFGgVYZXK5Iatlk2DrK4rIrkaFLFQqL+aLk6SQ2xvWM7HuGxrK18klggRqZQItiASALFpJo974+QyHvw1FQkAfWaHaZGagvIqdaiLeyvSv+dzRUXNK+9aTz5Y7j/a5AcU5la04ebtrWg3NqKzzFGspgI+KymjToTq3GvLfV4Olc03//KPmGxYLm1pYnE5xEcHLMmYz50RN2gXHZar9ViJejj1eev54b3XswbLtjAF399gA/f+CipbIFMvshUKseHbtjNNd97iEQmz31PTzR8nYeM59zafXITiatIVgrsltNitUmR0oqReDzg8ddWJHMpRlwkDCYGCXqDnNF1RnmMJDnEub3ncjRxtMzaigViZPJFQrXakBjWVqA4C5kihNqYzRXxGhXZFLKKSEsFFSPRlfnFvCqwtCFk1FvYW8hrtIZ9ZYpESkk6P0drKxOHA7c1lWll4pHvw83vVb///CMq/dobgNNe3vQp7Iokmc03VdW+FPB5rGB75ZjdWrB3HNDWVlulIgn6GEtY3/EJYxb6GevayjK3Hj4yzcbOCN3R6popn9fDP7/mLPrbQ3z+9v3c9JCV/OER8IHLt+MVgs/dvs8oiqz9Hh4aT9LZEjghOvguBC6RrBSUKZJFSgHOpZR1owPovmBtRTKXzr+LhKPJo6yLrmNtdC33DN4DqJ5ax1LHWNuylo5gh0kkyZxSJDP5Yu3/lB4PWU+EUCkFmYwikqkivoC2trLWdMRA1FIixRz4yzORIkbNQrKGItEzQwAy+RJSWl2Dm8JjN8BP/zdsvFjV4jTCnv+BW96v0p3/4B/gyVvg8f+Gc94E4famnzbk95ItlMgWimTyJTP2s9xQdSRSkXBOK5LGMRIwiMQ2HdGOaMhnzaMBJlM52iN+TuuL8aunRgBF/A8dmapbaS6E4INX7OCc9e3sOWbd2D3vlG7O2dDOHU+pJJGnhhM8Z0vt/zMHx1InvRoBl0hWDsoUySIRia5q13EPb2BlKZLkIOui6+hv6WcsPUaumGMsPUZRFlkfW09HqMO0tuK5ON3hbo7li6ypY4HkvC2ECmlTkaRzBXwBo+6hkLXmtfsj1nteo5ZEKRKjsaFtwa2MkVhjduegSFLj6mdmpjGRHLoTfvhOWHsevOl6ZeGtPReu+ETzz2dAX6O+a2+mYeNSIGCoynxRVs1rrwV7WrYZbHdQJJXB9s6WAKf0tvCDB3JMz+aYzRUZiWc5f2Pj7/plp/WacRU7dhodkJ8cmqlLJIfGUrzotJOolVMNuDGSlYJCRi1usHhEUqkyfMHaWVtz6fy7CJBScjRxlPWx9fRHVfuWkdSIGYBfG11LR6jDDLZb1lb9oGze10JEziIzM8hQG7P5Iv6ALdiuZ5HorC29vQJhv698wSqztspjJPNqIa8/m2wThXK3fUw1Y3zzjWYcaL7Q9tCoQSTH09oClR6riwybadoIKqYST+cRAqIVKjBmpP/qFN6JVJbulqDZnuTgWMqMj5y3sXklV4neWJCulgBPDte2oeOZPOPJ7EmfsQUukawcFLIQ6VKpnIumSCrIoUYld9m+y6RI4rk4yXzSVCSgguw69Xddyzo6Q52WtaWD7Q3SRAu+FqKkkelpSoFWpAR/0BYjKSMSwyKr0SbFHmyPVlhb2ULJvJO2iGQOd/ezRqC2UQFqqQTj+2H7SxaF5PV7NxrXRHL8rC2AQlE2HyOxW1uZArGgr2wmDChFIqX1mZiKxFjMD40lefjINEGfh9P75z9XRQjB6f2tdYlktQTawSWSlYNCRnVvDbU1TySH7oQHv1378XSFIvEGV4y1pTO21kfXs7ZFWTvDqWGGUkN4hIc1LWvoCHUwk52hUCqUB9vrEYk/SotIqzG7RsuVQMiwtqoUiSYSpw7AHtKGtdUS8JZVRFe2kp+XtaU/m1wDRZIYVnZc19b6+zUJvRiPJVRM7vjFSNT7mSuWTCJpOkZiZG1V2lpgG25l2FuTqRxd0QDrO8L4vcJUJGevbzPJbL7YubaVfSPJssJKO6zUX1eRuFguFLIWkTTbJuXBb8NvPlP78dmKGEmjYHsgZtk9SwzdBmV9bD19LX0IBMPJYQYTg/RGevF7/XQEO5BIjqWOUZRFW/pv7a9t0R8lRhqRnSHnUzZQ0K5IzBhJi1Wk6UCukYCPdL7omNnUWtEBeF7z2k1F0oBIzFb5pzR/7jqIrBBry1QkpZL1/jVpbemsLadsKXsr+WJJMjmbo6slgM/rYXNXC3uG4zwxGG8qPtIIO/tbyRVKpvKoxKGxFF6PmPtsmRMQLpGsFBQyaqGfiyIpZKGQrv14pcpoFGxf5kA7wLroOgLeAN3hbtPa0gqlM6QI8EhCDW1q8UXJFUqE6lQTy0CMPjGFkCWyBpEEQsZ/5LKsrfrWVshIk7W3kNewZpKou975xUiMzyabqL+fHt7Vta35c9dBeIVYWz6DSPIFW7C9SWsrky8aQ62qrz1mm9s+PZtDSugyUnxP6Yny2wPj5IqlBcVHNMyA+7D6/5ovlvivXYcZN1KOD40n2dgZMRMLTmac/K/wRIFWJMHWORBJBvKZ2o/PTqrzaZVRN9g+x86/C8RgYpD2YDtRo/ajP9pvWlvrY2p0b0dIXc+RuCKSkFd5zfUWHBlooUMossh41blDZTESQ5HoeSTgaG1FAl5yxRLTs9UdcvWdsLa2Uqa1NY8YSa4RkRxQ34vWOc4nqYGQqUjU9+Z4ZW1paytfsqytOWVtpQuOisQ+t33SKEbsbFE3DKf0tlAwhmmdtwiKZGt3CwGfhz3D6jO8/v4j/N0tj/P+6x6iUFRKZTXER8AlkpWDJVEkFb2z6rQEWW5FomtINPpb+hmIDzCSGmFtVCkSTSSH42oeedCj/lPWLEgEZNAKoM56FJFEgn5VjFkWbI/WD7b7rQW3ytqqmHuRnqsiKWRVmxZorEgmD0HHFlVQugiotraOb7A9X5yPtVVS1pZTjMQ23Grc6LPVFTWIxIhVrGsPm2ODFwKf18OpfTGeHIqTyhb4wq/2s6Y1xP3PTPKZX+7j6fHUSd8aRcMlkpUCM0bSPjdFUirUzsSqbMLYKP13GYsRdQ2JxtqWtQylhpBIc7u2tgYSAwD4hbKo6gVlhS09VhNJOOBV720xZy3gemY71B1uNRrPVi22lg+vg+1zJBIdu4LGMZKJA9C1OPERsBNkloDP07Dp4FLBb7O2zGB7oP5yFDRuINJG+m9lVTtYRJKwKZKuFsvaAjh3EWwtjZ1G5tbX7j7EeDLHl99yPm989ga+ctdBsoXSqgi0g0skKwdliqTJYLsmhXwNVaLbo2jUmFEOLGvn32KpyGBy0LSwALOWBDCJpC3YBlgxkmaIxBOyiCQpDEUS8KnAulYkwqve6zrWll5w1cyL6uppsFqZN5u+amLW1qOpXtZWqQhTzywJkUwks1Wvazlht7Yy+SIeAYEGWVRCCMJ+NW8klSvWCLZbRDJhtJDX1ta23iixoI8Xbl+8AsGda1uZTOX48p0HedmZazhvYwefuOoMTlujvoerxdpyK9tXCuxZW7kEFAvgbfDx6LYqhQzgkBM/O6lsEY1awfZSSTWKXKZixLH0GIVSocra0tDWlt/jpzXQamZ4edFEUnvB8YSs9yFOBMgppaBTn3OzKtAuRFOKBKia2dES8OIRlrU1myvg9YiGC6GJtF2R1LG2ZgbUtS1SxhZYr6skq1/XcsJSJMraCvm9Tc05CQe8ZnzHKdjeYgu2F0sSIaDDmGrZEvTxu49eTssitvvXtSiFkuTDLzkVUDc6X3nLs/jmb5/mnA2Lp35WMpZMkQghvimEGBVCPG7b9gkhxKAQYrfx70rbY38thDgghNgrhHjJUl3XioVdkUBzKcBzVSS10n+zM2ra4HLVkCSsGhINTSRe4aUv0mdu7wx1ki8pxeCRikhqdv8FfBGLSBLSFpz3BdRrzyUVkUDTRFIZIxFCqFYcNmsrEmhuIVQHGIok1FZOJFKWW10TB9TPRcrYgnI1d7xSf8Ge/qusrWaHgoX9XjPjzEmR+L0eQn4PyaxSJO1hv5khBoo8F2Mwl8bp/TECXg9vePYG0zoD2NzdwievPrNhbczJgqW0tr4FvNRh++eklOca/34GIITYiZrlfoZxzJeEEKvjE9AwFYmxEDYTJylTJBUoFtQ5wpXWloMiOV7FiA7W1pqWNfg81p1me1Dd0fk9fkpF9ZWot+j4w23m79MlVYhYpkjys1YrmlrWVi7Ftj1fwkd1w0aNWMhvKZLsHDv/6hhJ+8Zya2v/bfCvp8CUSi5g4pD6uYjWlt/rMW2l4xVoB/BVFCQ2u+CG/B5GTEXiTITRoN+MkXQ5dPddTMRCfn7258/n46/YuaTPs9KxZEQipfwNMNlwR4Wrge9LKbNSyqeBA8BzluraViQqFUklkUwcVB1gy46po0j0TJMqReJAJLqmYZmC7YPJQQSizM5qDbQS9UdNW0tDZ27FAjGyRgVxvUXHbyiSvDfCrNG7L+SzK5JUY0Vy4Fds2P05zhNKETjducdCPuKaSPLFuaX+amurfVO5Ihnbo74H+25Vf08eVNll0b7qcywAmoiPp7UVsLVIyTQxZlcj5PcyEjeIpAYR6n5b40Z7lKXGtt7YcUtaWCk4HsH2PxNCPGpYX/oWeB0wYNvnqLFtdaBUUouZjpFAtbW168tw85+Wb6unSNIO5FAr/XeZFclgYpA1LWvwV1TRv2jji7hk3SVl23Tmlm7YCPVjJIEW9f5lfDFmjTnqHo+wxUjsRFIxy11jVnXmbTfqUZwViY9kVle2F+Y2r312UhFEpLPcykqq1uQcuE39nDigplsuohUDlm13PK0trUh0+u9crC39PaitSHwkM3kmUzm6oyf3HJCVguUmki8DpwDnAsPAZ43tTv9TpMM2hBDvEUI8IIR4YGxsbGmucrmh7aZ6iiR5rLxmpFSyjnNSJLpNeaSJ9N9l7vw7nBouUyMan37+p3nHme8o26YVSdQftSbp1bn7C0QMIvHGDKVg7KtfexmRGAtRZdwopWIYbUKlCjv1oyqztuY6r33WiF0FW8sViSaSp3+jPtOJg4tqa2noRft4Wlv2OpJGHZ3tsCuXukSSLTCRzC6LInGxzEQipRyRUhallCXga1j21VFgg23X9cBQ5fHGOa6VUl4gpbygp+ck6fOvFYVdkVQRyaiqGSkZDeLssQ67IimVYPf1cMPbwOODru3WY96g82hZs7ljfUUykhopm60+X0xnp02l0QgdQcvaaqa5n64jSXuj6k63kkjKYiQ1rC1DkbShFYmztVVGJHOxiWYnlFIMRFVdi/5MkyNG4WQGDt6hxuguYqBdQw/gOq5E4rHmkaTzRbPivhHsn30taysaUhMsp9N5s4bExdJiWYlECGG/DX0VoDO6fgy8UQgRFEJsAbYD9y/ntR1X6DtiX1DdpYIDkajpbhgZTGXkYVck338T3PI+Fch99+3Qscl6zGctnD946gd8d8931d9akYTqpyp+ctcn+bvf/l2TL6o2JjOTtDd4Lg17jKQZawt/mAIeZkULs7mCpRQq03+hdvdfI6uqXSuSGtZWwtb9d07z2nU2XdDI8tEB99QYbL0UfGH4/ddUJt0ipv5q6Nnnx5VIfMqEKBRVHUm4wbx2Da1cPAJaasSlYkEfA5Npo8+Wq0iWA0v2TRJCXA9cCnQLIY4CHwcuFUKci7KtngHeCyClfEIIcQPwJFAArpFSFp3Oe1LCrkiCrYAoJxIpLdujmKu2qPTx+TTs+wVc8E648rPVbTV0TKCQ5WdP/4yp7BRvPv3N6tyh9oZ1K2OzC7cSS7LETHbGVBqNUBYjyTbRblwIUkRIeaIqRqIXG6f0X48XENWKJKUVST0iUdaWlHJ+1lbnVmtIVTahsvWSI2r07pYXwH4j4L4U1tYKiJGUtUiZY/ovKFurchaJRjTkM9Wrq0iWB0tGJFLKNzls/kad/T8NfHqprmdFo2CLkXg81Y0bc0mr/XmxjiLRx/Sd6dybyTYRMFvMcjRxlGKpiHdif1MWSjwXJ+RdWI+iRC5BURZNpdEIpiLxx8gmiwhhtcqohRv9V1GInkM6V7SUglP6ry5KrGFtdXkUkTg1NowGfRRKkky+VG6hNYPZScvaAvX5FvNKCUV7ofd0G5EsgbW1ArK2tLWVK8o5vX96P6caEg078bsxkuWB2yJlJcCuSKC6TYpWI2CRjl2RVBJJyKqlKIPPUiTZYpZ8Kc/I7AiM7YWeUxteZjwXJ1eq0fSxSeiJh7o+pBG0IokGomoWic/TsKDsxsibeDh4QblS8AXU+1TIWAs4OLeNMeo8OjyzNftRmY0bs/m5KZJiXhWARrpsiiSpbC1QRLL9xer3UPuSJECsiBiJzdqaWx2JViS1rz0atEjGtbaWBy6RrASYisROJDZFYicSffdsVyQ6mytt1I7Uij/Y0l1zxnkGxvcoS6V7R91LLMkSyVySbK15Jk1iOquusdlge3e4m+eufS7P6nuW2UqjEUJ+j1IK9voEb9CKBQVsg4a8/nJFIqVpbbWLZJ1aBT3cqkB6LnUk9gw5k0ji1mcc7YOOzdB9asPPZL6wYiTHMf3XY8/amoe1Vefa7Qqyy1UkywK319ZKgKlIjIW+ikhGrN9Na8uuSIzj9THhGkSig+2GIgE4cuwhLgToOa3uJSbzSSSSfK2mj01iMqPu9psNtvs8Pr764q8CcEP+kbqpvxpBv5dMvlgebPcFLcII2BrpVVpbuaSZEddGsuZiq+/mdd+nOXf+DXdUW1sALb3q5+u+tej1IxorI/1XvbZ0vki+KJtWJGGjQ3Bda8uw7ISA9ohLJMsBl0hWAqoUSStM2+ozGyoSTSRakdSwtkxFkrUUycQeta2n/t1v3CiQXDRFEpy7ZZMplJry0kN+LzNpbTkZX3GvbUHxVxKJjRx1H6xAlFg+VXOx1fEF3fepaSLRqdaRLitrK5uEkvHZRQ0i6Vu6lhsrwdoSQuDzCDOFeu7B9nrWlnqsMxLAWyMg72Jx4VpbKwFzUiSaSOrFSBookmLeUiSJAUUw7ZucjzGQMCb5LVqMpElFYkcmX2wYaAc1+CqbL5a33vDZkgTsisRXoUiMYkS6TiEqk2zvdp63rZWKbtcRbtba0kSlCxJBZW3pz1gTyRLCUiTHz9oClbmliWSudSTNWFtuoH354CqSlQDHYHsja8tBkaSbVCQ2a2sgOwXd241U2NqI55QiKZQKlGQJj5jfPchUZoqQN0TYF57zsZkmg7LhgJdEpkC+KK2sLZ9tUSmLkVQQiZGxRdc2PMOP8JmrbG34bdB38yNzVSSzNkViWlsJFWwPtoF/7u/LXHH1uWtpCXqPa9YWqDYp8bT6Ps+1sr1WVTtYisQNtC8fXEWyEmBaW8YXP9qrMnv0WNgya6te1ta0Sm311fgPZCieQmGWolGmMyCzyCaCugnbbPFcrXG9TWAqO1Uz9VdKaU61c0I2X6pfjGgg5PMyNavOUxZs1yjL2qoItmvFYKTdCm0XVkDfEY/MOUZinD/caaR7+y1FEl2eTg2bu1t49yVbl+W56iFgUyRzD7bXJkFN8m4NyfLBJZKVgEpFom0mHSdJjkDAyPCpjJEE28pjJLXUCJh1JLmcqklZ29JPWsB4x8aGl6gVCSwsTjKVmaqZ+nvrEyNc9E+/YqoGmTSbJhrye2zjb3VBom1R8ddRJLpHmW4to7OsKqDtk9G4JpJms7Ym1ecciKhocDCqYiTJ0UXv8rvS4fd6iBvdAcINxuxqaCJpi7iKZCXBJZKVAHtBItiIxJhLkRyFdqMVWaW1FW4rj5HUiz0Yd+VZY2759vAaAI5EG8crFkuR1OuzdXAsSa5QYizpTFSZfLGprC072VgtUuzWVr1g+7ja1mbMSqlBJF6PIBLwMhLPcooYZN2hGxtelzr/pLK1zGuJqayt5MiyxEdWEnxeK9jebNaWJpDuOnNGoiEfAa+HNW0LK5510TxcIlkJqFIkhkKYOqwa+qVGrYWtMtgearcpkpn6isSniUQRzzavujMf8De2AGayVsxmIQH3qcxUzUD7WEK9Jr24VCJTaK4C2j5Bsaxpo0YZkfirg+2RbquBZQ1rC5SFMhLP8Be+m1h7z19ZzRfrQVe1mxcbM6ytMSv1d5UgYFckTRLJzv5Wrv+Ti3jeKd019wn6vNz4vot560X1E0hcLB5cIlkJ0KSgffxoryKV6cPqjrhUcCASrUjaLUWSnq5dQwLmXXmuoKytzbkcPik5IhtbVXZFsiBrKztVs8/WhGFp6WaIlcg0GyOx7dOcIqmIkUS6LCKpoUhAZT0VCnle4HkUIUsqaN4IleOPg1Flp2VnVqUiSWaNGEmTMSYhBBef0lWzz5bGORvaj3tW2mpC00QihHi+EOIdxu89RpdeF4uBQkYtaLo/lhBKlUwfVmoEbERSUZAYapu3IokkR+mXgoGUY8f+MthjJPMtSswVc6TyqZrB9nFDkejFBeDhI1Mmsaj03yasLZ+DtWVXJPaMMSdrq6XLIstiYCQAACAASURBVOS6ROLjWWIfrcLog5aurV6s80+UE0kgqiYhwqqMkUhj6tCcBoO5WHFoikiEEB8HPgL8tbHJD1y3VBe16qDntdvRvklZWzr1t22DtS8Y5BNUgWPd0LFhsN1QJEVFPMGZYTZ6oxxJHGl4iYuhSBr12RpPlltbmXyR13/1d3zvviPm380F223Wlr8i2O5vKW9oWWVtjStry2e8t5ockmMw9UzZ88RCfi7z7rY2VLb+d4KTtWXvs7WK4PNan4NLJCc2mlUkrwKuAtVXW0o5BMSW6qJWHfS8djvaN8L0ESv1VxOJPUbiDykCymeUP5+J1w+2a0Vi9OYKJIbYEOljID6AlI4DKU3Ec3G8Qv1nn2+wvVGfLW1tJQ0iGU9myRclE6kcxZI0WmnM19oy3t9ARYFhlbVlC4aHOywi+emH4JsvKxsKFgv6uMyzm5Q0zt2ISEpFpXDswfagLRV5lRFJwGvZU80WJLpYmWiWSHJSrTQSQAjR0mB/F3OBkyLp2KQUxvh+9XeVtZVRx/jD6vdsHJANFIkmEnXnHywW2di2lUQ+YS7ytZDIJegKqQVwvkQyla2tSArFkln7oa2sSTNmUjDH7DZz5+qYtaWJOlDx1bVbW4WcilW0GIHccIdlbQ3cD4khePou89D1nglO8wxwh3i22tCISDIzgKyIkbRav68ya8vncRXJyYJmieQGIcRXgXYhxJ8At6NG5bpYDNRSJABHf688fb342BWJL2goknTjho2gLB2Pz7K2pKSnSxUjTqQn6l5iPBenK2wQyTyztrS15aRIJlM50y9PGDGSiaRFLJkmxuxq2BVJuDLY7nciEuP1mO1LDMUQaldkHh+CpDFi+NEbzEPPzdwHwK98l6oNdTK81PltVe0a9uLIlpNkdHST8BvtbrweYQ66cnFioqlPT0r5GeCHwE3AqcDHpJRfXMoLW1WoFSMBGHxQWR5mw0UHRVLKW4tgPUUC4A2YMY6AhEiHqnBOF9L1jiKRS9AdVnfqC46RONhv9toRHSMxra5sgUyhiTG7BsqD7RUxEkdFUoNIwu1KkQw9rP7uPQP2/I/ZceC0xC6OlHo4GDpdPW5XJBMHYff15c9lr2rX0NZWuNMaPLZKoK0tV42c+Gg22L4FuFtK+X+klB8G7hFCbF7KC1tVcFQkBpFk44pIKsfCakWiezPpoHyjZojeAFnjHMG2dYSNNNd6RJItqt5cmkgWYm0JBG2BarLT6gOsGMlkygq+z0WR6DqSgM9jdX/ViqQqRuJXlhZYfbYqra3Bh0B44cV/r4oH9/4cjtzHhun7uaN0LjIQo2o88kPfhlveZ50bbJ1/bVlreibJKrO1wLK2mi1GdLFy0ayevBGwV1sVjW01IYT4phBiVAjxuG3bvwohnhJCPCqEuFkI0W5s3yyESAshdhv/vjLXF3JCw0mRRGyjWKN9trGwtqwtX8hKZU0Y1ksjReILktOKpGs7EZ9aWOsRic7YakQkx1LH+Is7/oKUUTlfianMFG3BNrwODSJ1xlZXS4BEVqkue/A9bbQ8aSr911AtZf2vTEUSLd/Zrkh0e5SIJpJ2FWwfegh6d8Ipl0PrOrjrX+A7V5MO9fGVwlWEA4Hq8cjaxrI33HRSJLr1zSoLtINlbTXbHsXFykWzn6BPSmmuHsbvjRrZfAt4acW224AzpZRnA/uw0okBDkopzzX+va/J6zo5UMxWKxJdSwLWIuMLlteR+EIqcwusBatejATAGzTrSILdp5ldeGeNIkUn6FkkOkZSy9r63dDvuP3I7eyd3Ov4+HR2umHq7+buFkuRJC1rK1vQiqTxV1ZbJRH7na62Bv0OWVuyqDKqKmMY4Q41ffLoA7DuPBVjOut1ML4Pek7l9y/6PsN0qThMqK28jkQH6e1EoonKHgvR1tZqJBKPa22dLGiWSMaEEFfpP4QQVwPj9Q6QUv4GmKzY9ksppa422wWsn8O1nrzQ6qIS2t7Stoe95kHbYaYiGVY/GyqSALmEKkAM9p5uEkk9RaKLEbUiyZecCxKPJo8C5e1U7JjKTNVM/R1P5gj4VH+kyhhJPJMnk1eCeC5ZWyFHRVIZIzHiEsW8YW0JK7FBV7dn47D2PPX7cz8Al38c/vgnhNrV5xIJeFXPM7si0aSilSKoehFfuPwaVrG1pQPsLpGc+GiWSN4HfFQIcUQIMYAqTnzvAp/7ncDPbX9vEUI8LIS4SwhxSa2DhBDvEUI8IIR4YGxsbIGXsEJQcFAkUK1I7DZMpSJJjIDwWFZJLXiDZI0MrUDf2UT8ja2tSiKppUgGk4MAzORqEEm2duff8WSWnmiQ1pDPytqyBdu1tTWXgkRna8sh2A7qfU2NK/LQ1ps93rT2fPWzpQsu+RAEY2YLDqVI2iuIxFAkmuBBnb+lp3yEbmD1KhKfEWx3YyQnPprqfS2lPAhcJISIAkJK2URTodoQQvwNUAC+a2waBjZKKSeEEM8CbhFCnCGljFceK6W8FrgW4IILLqhfRXeioJYi6XBSJPasrQpFEmwtr9p2gi9ANg9eKfF17yBsrGmz+drWlo6RaDVRK0YymDCIpI4iObv7bMfHxpM5uqMBYiG/rY5EEZaUVk3JXNJ/I37b19sbUGottqZ8Z5NI8lafLQ2tSLwBFSOpgJ570RLwQakNJg9ZD+pU4DJra6x65ki0TwXyO4//fJDlhqlI3GLEEx5NEYkQIgi8BtgM+IRxRyWl/ORcn1AI8Xbg5cDlRpEjUsoskDV+f1AIcRDYATww1/OfkKilSPTAKa1MDEVSLBXxFLKIMkVyrHF8BMAbJCcEAQT4AvilxCu8TSmS1kArQW+wNpEkaxOJlJLpzHTdPltr2kJEgz4y+RL5YonJZI5IwMtsrmimB88l/bdsgRIC/vReK5CuYVpbOUUkLbbHNZGsOctxWJieSaKUTy1FYre2RlWw3o7WfvhfD0D75oav62SD303/PWnQrLX1I+BqlIpI2f7NCUKIl6JssauklLO27T1CqP4bQoitwHbgkPNZTkLUUiTbroB3/1otZKCIpJDlNT9+Dd8KFMoVSWq0cXwEwOsnIwRBoRZBIQQRX6SprK3WQCsBT8CxIDFTyDCWVlajE5Ek80kKslDX2lKKRF3XRDJHKldkY6ey3nSL+WYUiccjCHg91VML29ZbxKuhCVxbW2WKxLhWHR+pQGvIj98r1IwM+3jkfMbqf1YZbG9xaH/eubWxkjwJ4cZITh40O7R5vZSyMgOrLoQQ1wOXAt1CiKPAx1FZWkHgNkPV7DIytF4AfFIIUUClFr9PSjnpeOKTEbUUiRCw/lnW394AmUKWgzMH2eOR5TESWWqOSHxKkQRtbdXDvnDDrK2wL4zf6ydgK2i0Y8jWQdgpRlKvqr1UUiN2u6NBc7rd4Ql1n7K5q4WnjiUsRdJE+i9A0O9pzjIps7bGYcNzrMdi/bD+2bDzasdDAz4PP3jvxZzSE4X72lSNSbFQXuGuFYmUytpaZdXr9aCJxO2zdeKjWSK5VwhxlpTysWZPLKV8k8Pmb9TY9yZU1fzqg5S1FUklvAFGi0o5jHuk2SLlIz1dnJrL8c5GxYiggu1CELS1Ug/7w6Tz5YqkWCoynBpmfWw9iXyCmBHEr2Vt6fiIT/gcFUm9Plsz6TyFkqQ7GjQVyeEJRWybupUi0S3mQ03WHDx7cyfnrG/m/TCsrUJGpf/aFYMvCO++ve7h52807C9N4pkZy9YKtlqKJDOt5sq4RGLCtbZOHjSrp58PPCiE2GsUEz4mhHh0KS9s1UAvyk6KpBLeACMl1SdrzOMBXwjpC3FHJMzd4XCTiiSgYiS2wjwna+vWZ27lFbe8gtHZUeLZOK0B1Vww4A04E4kRH9nWsc2ZSOooEl1D0h0LmplQhyeVItnUqbKsxpJZhFBT9ZrBN//42bz9uZsb76gVyeM/VPUknac0df4qaBssM22l/vacpro3FwvONSSrHKYiaSLu5WJlo1lF8rIlvYrVjMoxu/Xg9TOanwEPjPm84AsyVcqS9ng46veR9bfSkI68QbIeL0EbkThZW0OpIQqlAnsn95LIJcqIxNHaSg4R8AQ4pf0Udo/urnq8mT5b3S0BWgxr6xlDkWzusmIkIZ8XYU+dXQxoRXLP51WK79lvmN95nBRJz6lw9H5laemZI04xklUKnxsjOWnQbNPGw1LKw0Aa1UrebCnvYoHQg6qaVSRGg4GUx8Osx8tgRoWSRrxejmSauC84/23kurcR9FrPF/aFqxSJztQ6MH2AeC5uWlu1gu1Hk0dZG11LR7DDUZFMZFTtitOYXd1nSykS9RqOGESy0SCSRKawNHeuWpF4A/Cqr4C32XurCphEMm0RSa/RzDF5zEYkriLRCLh1JCcNmm3aeJUQYj/wNHAX8AzlxYQu5ou5KhKsMbTjssCgkSklheCpfLHWkRY2P49spIuALdge8Ueq6kh0WxRNJM1YW+ui62gNtpLMJ6uq34/Ej9AT7jELIO0wra1o0EypfWYihd8r6G+zYjlLsuDoFN/LP6YUxHyhlVaZIjlN/UyMWAPKWlZf4WEt+Nw6kpMGzd7ifQq4CNgnpdwCXA78dsmuajXBVCRNEIkvWEYkozLH0dSg+fe+XHPt3bPF7PwVSQMi0Z197aN5AZ6JP8Om1k2O1zOezOL1CNrDflqNGEkiU6CzJYDXI8xMriWxQPrOhPffBxdfs7DzVFpbwmvVASWP2RpCdjkfvwrhpv+ePGiWSPJSygnAI4TwSCnvAM5dwutaFTiWOsaPD/9S/eFt1ANT7TMiJL2GPTReyjKUHMJnTIQ6VKc63Y5cMVemSByJxFAkB6cPkswlaTUm+TllbSVzSWayM6yLraMtqBbUSnvrcPxwTSKZSObobAng8QiCPg8+o5lfZ4siO00kwaVYcISA3tPK25bMB3YiyUyrv3VHgsSIsrbCnfO3zk5CuFlbJw+aJZJpoz3Kb4DvCiH+HWy3xi7mhVsO3MLfPP4VZoVo2toaEZKdrVsAGCtmGEwOsi0vCZdKHC0km3reSkUS8UWqgu1akWSLWSSSmL+2ItEZW+uizkQyk51hMjPJ5tbNjtejihHV9fz/9t48vI3qXPz/HEnWZlve43jJvpA9JglkgUAgkKSUQNhaaG8LtMC3lPyg0N6ydANaWujNLW0KvZRSCm3vhVC2trS0EPYtkAVnX0mcxLHj3ZZs7dL5/TEzkmzLS2I7tpLzeR49kkZnZt4ZSfPOu5z3FULE4iR56ZqyM94P6eweazqYLFrGlq9Jc5lZrJoFYsRIVHykHWoeyclDb/+Zl6IF2m8H/gV8BiwfKKFOFYy4hNtk6lWwPWJKo94EE5zDsUYldREvR1qPMDwiKAmHqYt2LoH23t66WO0qg2Ak2Mm1FYgEiETjMZaWQEu7C79hkSTL2jKq/pZmlMZcW4YiAi0+AnRpkdTpdbYMjDhJrq5IjPe9nYw4KAgRn91uKBKAjOHapESjYKMihnJtnTz0NmurTUoZAZzA34E/o7K2+oxhBbToc0J6okFIIkJQaMmkIBKhJuShqrWKYREzpaEwraYWwpF4/7G2QJhrn/yER97a1247gWigk2sLwK/3cgdNEZw+LF4axAi228y2TllbxmTEriySCncFAKOyunJtxS0SgEybFifJyzAsEu39kLZIILkiySzUFUmdSv3twKg8J06rmRG5nRMwFKlFb7O2/p8QogbYglZIcSOnSkHFAcSIS7jNvbNIjED7MIuT/EiEna2VhKIh8iMWSsNhomnNVLfEYx1tgTBRCWt31LTbTkeLxMikMiykSDRCa6iV4enDKcnQigwawfY0U1pS11Z6WjpZtqykiuSg+yAmYWJExoikx9UWCMfcVxC3QGKuLZvh2hrid672rHj6b6JF0lqj1UJTFkk7JhZmsuP+ZZRkO3oerBjS9PYW7zvAVCnlaCnlWCnlGCnlqVf3up+JKZJeWiQ1ek+wQrODgkiEA16t10VuxEpRSCJMIXbWxrO4fHo68Gd1bVTUazPFpZQEIsktEkOexCKN47PHx15D8mC7kbElhCAjLQOBaFdv66D7ICUZJaQZk/864A9F2ykJQ3F0DLYPeReI0ZPE1xyf6W5YJP6WU7LniOLUoLeK5DOgdylBil5juLaSxUiklPz3a7s53Bg/7cZkxGFYyI/E4xmusI2csHaR3l57ILbclzCvZO1OzSoJyzBRGe0UI4G4IjGUgMvWWZF0FWw3LBezyUymNbOTRdJVfERKSSAcwWaJ/xQzO8RIjPcDkrXVn9iztHpd/pb2FonUvwfl2lKcpPRWkdyNVrjxt0KI1cZjIAU7FTAKJSazSOo8AX795j5eS3BL1Ub9WKQkNxJlWDiuJKwRF1lSu9Dvbz4U334wPuaNndqEOEMJdMzaggTFpqf+ZlmzWDp6KUtHL6XAqbllrGYrYRmOBeallO0UCUCWLSumSKSUVLgruszYCkclUUk7RRJzbWV0CLanQoykpRKQCTGShEZayrWlOEnpbVL7b4E3ga1AtIexil7S3rXV3iIxrAl/glVRE/FTEIlgCnpiFskwxzD+YP86w9PDIO/jiJ5BBXFFMmtkNusrGmnxhYgILeOqnWsrTbdIDMVmNLKyuZicN5lV566KjTUUUDAaxGFy0BRowhf2tVck1qyYVVPnq8MX9nVpkRjH1861pQfX4+m/aZ3GDEkc2WBktClFojiF6K0iCUsp7xhQSU5B2gfb21skhiIJJCiS2oiXwnAEAm4KdEVSnFHMkXAWWekuzAEX9YGaTtu4eEYxmw4188G+emaN1esbmeP76+jaSuyI2BGrSbu4ByNBHBZHu4wtgyxbVsyqOeg+CHSd+hsIa/cliRZJtsPI2tKUVizYPpTTf6F99eWYa6swvkwpEsVJSm8VyVtCiJvQUn9jkwhOqeZTA0B3MRLDmkiMc9SG25gYDkPAQ4Hu2irJLGFfKIIzzYwjnENrIP6VGOtOK8nEnL6LI02TmBrRvvJ2tbY6uLYMt5SRgZWIsZ7hIotNRsyMKxKXzcVhz2EgnvrblWvLsEhsCUriqjkjGJ2fTpauUIwYiaOXvUgGjURFYtTeameRqBiJ4uSkt4rkS/rz3QnLJKAyt/pA3CIxg6n93XbctaXdsUspqQm3cnYkoikS3SIpySjBG4zgtJpxhXOp9B0lGpWYTCKmjKoCW3COfIq9LeM5N6JVpO0u2N6tRaIrEmNSYuJkRAPDtVXrreXfB/6NzWyjML2w07YgwSJJiH/kpltZOjV+AY7HSIa6RZJQIt+wSNIcYMuCsE9rdKVQnIT0qEiEECbgP6SUqkhjPxNXJJ2/Bn+HGIkn5MEXDTE8rCmSHJOV78z5DuePPJ9fB7fhsFrIi+ZT6d1Do1drW2us2xDQ0oTrfS0Ewp1jJMY8kpg8ATd2s73dGIPEGAloFkmOLaddVV8j2H7xSxcTjoa59fRbMYnk1kQgZLi2ulYSsRjJkHdtJVEkoKUAB9v6Xs9LoRii9OgrkFJGgVU9jUuGEOJJIUStEGJbwrJcIcTrQoi9+nOOvlzo2WD79C6Ms45nn6lCKBIiHNXmhbhNnb8GX1C7wPr1O/Z6r1Y9tkC3SITFxrVTr6XIWUIwEsVpNZNjy0OY22jxacrCGzQUidY33O1vjVkSiRaJES+JlWxJKBvfkcQYCWiz2hPjIwClmZp1cnbJ2fz10r9y3bTrujwPft1FZ+smI6vQZcNsEhRm9aIe2WDSLkaSoFSyRoCr+MTLo1CcIHrr2npNCHEF8KKU8lhKozwFPAL8MWHZXcAbUsoHhRB36e/vROvCOEF/zAX+R38+KUkskug2db5T7Zi11RzQ2rdmR6MQcIPujvLqnzutZvJN+QghqfLUMa7AFdtGnU9XJMG2pOm/ZpMZm9kWn0cSaInV1upIxxhJdVs1E3ImtBuzfOxy5hfN79KdlYhhkXRnbRRlOXj/zvMY7koRRWLNiHdeBPj8Kq3drkJxktLb6OUdwF+AoBDCLYTwCCHcPa0kpXwX6BiQvxR4Wn/9NLAiYfkfpcY6IFsIUdRL+VIO46KdjQW3EHTUzx0VSSwAHomC3x0LzhtxEIfVTIFTC+ZWeepi27BZTFS3aa6ttqA3ZpF0dFslVgDu1iJJiJFIKTnadpSi9PZfk9lk7pUSAQj0wiIBTZn0e5vd/sawQhLdWgC5Y6Fg4omXR6E4QfS2aGOmlNIkpUyTUrr098cbOSyUUlbr260GjLoRJcDhhHGV+rJ2CCFuEkJsEEJsqKurO04RBh9DkRQKC2FBp34g/qCR/qvdscdmm0c115aRLmy4rxxpZor0VNMava2rPxjBYTXHFIk37CUQ7ezagvY9SdxBd5cWSSxGEgnSEmjBH/EzPH140rG9wd8LiyRlMM6Zo3NfeoXiZKbX+ZR6u91V+uPiAZAl2e1mJzealPJxKeUcKeWcgoLUzcuPKRKpXUATy65DgkWi37HHZptHo7oi0S7o3qDmMnFazZRkajq5zlcb24Y9TVLn1RVL2Jc02A5JFEkPFkkwEuSoV3OZ9UWR9NYiSQnS7JqC72iRKBQnOb2t/vsgcBuwQ3/cpi87HmoMl5X+rDezphJILA9bClQd5z6GPEZgu1Bq+rNjR0FDkRiuq5ZgCyYEGVEJ0VDMIom7tiyMyNIUSYOvQdtHMILN5kHq+jgqArgDmrLoaJE405zt5pEkm0MC8WB7IBrgaJuuSJx9USSdJySmNPZspUgUpxy9/fdeBFwopXxSSvkksExfdjz8DbhWf30t8NeE5V/Vs7fmAS2GC+xkxLj7H64XnOlkkQTbWyQtgRYy09LjX1jMIokH2/OcGciInaaApkj8oQhmW1xBCVOAFl9yReKwOPCFfISiIXxhX5cWibFeKBKKK5K+WCRJSqSkNAtWwswv9TxOoTiJOJYG0tnEA+fJb1c7IIR4BlgE5AshKoEfAQ8Czwkhvg4cAq7Sh/8TTTntQ6s0fP0xyJZyxFxbkSiYOysSf4cJie6Amyy93S2QNEZis5iQ4UzcIe1r8oUimNO0bC+LSCNkCtLi16yOZK6tWm9tzIXWlSIxSsEHIppFYjFZyHPkHccZ0DjpLJIF/99gS6BQnHB6q0h+BnwqhHgLLZZxDu1nuSdFSnlNFx8tTjJWArf0Up6UJ6ZIwmGwxmMgsc87Zm0FW8iyJioSPWsrFI+RCCEwSxet4Sbts2AE7Nrr4c5SKjzBmGvLcFHFkFZqW93tCjYmIzHYftR7lEJnYZeTDXtDXJGcJBaJQnEK0q0iEUKcpc9ofxF4GzgDTZHcKaU8OvDinbwY8YjCkDYfoyvXViDRIkmMW8RiJNrnTqv2VVpkFr6oVrbEF4oSSW8k35FPti0HTI14An4sJgvmDiVZKhsi1LV5qG3TrBmj93pH2gXb2zRF0hf8oQgmAWnmIZ7aq1AouqSnW0mj58hHUspqKeXfpJR/VUqk7xgWybBQEEHXWVvBSJRIVGoWSRJFYmRtOayaYrCJbAKyGSklvmCYsGikOL2YTKsTIUK0Bf2d4iMAwZAFYQpyqFlTJD1OSIxqiqQv8RHQLBKbxTz054goFIou6cm1FRJC/AEoTdbISkp568CIdfJjKBJHyE+mSO/k2krsQxIIR/TZ5omKpP2ERKeuSJzmHFoJ4g17dWXUQFHGTJACYQrQFkquSAJBC5iCVLZopVi6ipFYhAWTMOEP+6nx1vRZkfhDkZMj9VehOIXpSZFcDFwAnA9sHHhxTh18IR92sx1TuA6XKa1LiwSgNRDEE/SQZUtIKzUsklAEi0mQZtYuxhmWHGqBOm8d3mAIk2ygOL1YSx82B/GF/FitnYsxegNmhDPKEY+Wjd2VIhFCYDVZqW6rJhwN990iCUVPjsmICsUpTLeKREpZL4T4C1AspXy6u7GKY8MX9uFMc0DYj8tk6zJGAtDocyORZNkTFUncIjHcWgAuay6EoN5XTyDqwUaI4enDCUVDCFMQXzhAuqNzzao2nwmcUN2qNcbqyrUFmnvrkFtr6duxPMqxEggri0ShSHV6U/03Aiw/AbKcUnjDXhxmByBxme1J0n+jWHUro96rpfBm2bPBpOv+hBiJM0GRZFu1VNyatlqi9j0ATMyZiMPiQIoAgYi/U+qvPxTBF9C22+CvwWlxkmZKoyusZiuHPJoi6btrS1kkCkWq09tbwQ+FEI8IIRYKIWYZjwGV7CTHF/bh0GMVLosjafpvtlO7mDf6NEXisrrAiG8kTEg0MrYA8hxa4cbqtjrScj4m21LMrMJZWr8QIfFH2jrFSOo8AaTUlEtLqL5bawS0FGCjGnFfZrWDskgUipOB3s4jWaA/35+wTKLFThTHQaIiMUesuMO17T8PRhiZ66TWE6DRn9D61pwGIdqVSHEkzArPs2cjpZmPqtZhcVZQlvM1TMIUa6cbohWrub2iqPX4kVFNafmiDRRbu0/pNawVu9neZSmV3qJlbSlFolCkMr1SJFLK8wZakFOF9/fW0+QNajESXZHsORLAnedGSonQS8onWiTNft21Zc0C3S216s0K/l9ZKNZm18DlsCLDGXxS8z4yauaMvCVAvJ1uVLRiM7W3SGrcAYhqyyLCTZat+5LnhkUzPH14n9N2/aEI6bZjKbCgUCiGGr0t2lgohPi9EOJV/f0UvcSJ4hh5/L39PPr6dryBFhxGrCJiIyzDsZRgY7Z3jlN3N+kFHV02V0yRVLVKDtS34Q21D7Zn2CzIsDYDPuyZTr4zF4i308XchrlD/KPW7UdGdVmEJDOtZ9cW0OueI92hLBKFIvXp7T/4KeDfgNEvdA/wrYEQ6GSnsS3AZb7n8TXsxRHRM7MimrVgBNyNjK2c9LR2yzWLRK91hZWqZj++DsH2DHtckYSa5sbcXoZrS4goQnZQJJ4AZuIBeKspvdtjMOpt9TU+AvEJiQqFInXprSLJl1I+B0QBpJRhINL9KopkNLYGmRjeUEjpXAAAIABJREFUiw+Jo34vALKjItHnkGTrFok76MZhcWgXcKNDIWlUNfvwhdoH2zNtFsLeMRTbZhDxjY5V1Y1ZJICU7V1JNe4Aec54HS9T1El3JLq2+oqakKhQpD69/Qe3CSHy0BtNGWXeB0yqk5hGb5AxVOIzmXC2HAFARjVFYriwDEWSqyuS1lBCnS1LXJFUt/jwBSPtSrBn2C2EGs9htu0uQMTcXoZFou2vvSKp9fjJT3clfO7o9hiMgo/9oUiURaJQpD7H0rP9b8BYIcQHwB8BVS/7GPEGwxDyMYJavCYTDqNPe0CLY+xq3AXEXVtGsN0b9sSLKBoWiUyjqtnfKdieoQeu6zxaJ0RHEoskEml/4a51ByjMiFskoVDnCYuJGPNQ+kWR6H3lFQpF6tLbf/AO4CVgPVAD/A4tTqI4Bhpag4wTVYRNkijgyBkLQDScQ661hA+rPgTiFokRbPdGEiySBNfWkZhrK64YMu2a8jEUiTOJRRKOdLZIhrsyYuXg/YHOtbgSiSmSfoiR+MPRk6eplUJxitJbRfJHYBLwU+DXwATgTwMl1MlKkzfIBHEEn37Bdkxcxg4xgUNyGCPsZWw4uoFgJBizSLKcaQgB/khrgiKJB9v317UiJe2ytjLt7S0S4yJtpP8ChELxrz0QjtDkDVHocsTGtPm6ntUO/RcjkVISVFlbCkXK09sE/tOklDMT3r8lhNg8EAKdzDS0BZloqsStn3Zn3kSulg/gJkyhZSabI/9gU+0mfKHRQLzroT/aGiuiKM1WBBASabj9elOrhDt6m8WExSSoa9VdW9bOisTji8tkKJxhmTYcDQ7aQm2427pXJFm2LPLseWRYM47/ZBBPc1YWiUKR2vT2VvBTPcAOgBBiLvDBwIh08tLUFmSCqGSfKADAZrLjCWjKwGWahMVk4cOqD2Ml5B1WM7Y0E8Foa6xsSVhXQsNy4jPKE7O2hBBk2C0EjYu0frdvNpmxm7XYR2VTKDa+VlckhS57zP3V5On+/uLr07/O05/rew1Po2mXskgUitSmt//guWj1tiqEEBXAR8C5QoitQogtx7JDIcRpQojyhIdbCPEtIcS9QogjCcsvOsZjGfI0tmmurd1imL7ESizeHrRw+rDT+fDIh+16jNjTAkQJk2fXijGG0KyFMcPjfdITXVsQD7hbzSYs5vhXbATcW9qgsknr0Fjr9gNQkGmLWS31Ld3/LFxWF6Nco47p2JMRCGvHqdJ/FYrUpreurWX9tUMp5W6gDEAIYQaOoAXyrwcellKu6q99DTVa3G5GiloqxHSggWjECmgWgTcQZsH4Bfxq06+oz9KaSznSzKQ5tbLu47PHAxCQZpzA+OI82K611HV2oUg6KhhDUUhpYX1FI6U5znYWSSxG4rfS1BYkJ71z35L+xB8yrCbl2lIoUple3QpKKQ929+jD/hcDn/VxG4PObW/expuH3uxxXFrTPkxCchAt3TcSjsciWgMRFhRrtTH3ejYBeuzAWg3AhJwJAPij2kV3Ykl+bN2OCsOlZ245OsQeDIvEYbbxyYEmQEv9NZsEeelWnGlOnJYMwMS/tw98N2VlkSgUJweD/Q++Gngm4f1KIcQWIcSTQoicZCsIIW4SQmwQQmyoq6s7MVJ2QyAS4M3Db/Ju5bs9jnW2aDPZK4UW3wiHNctBCGgLhJmUO4lcey4HvZ8ihBY7iKZVY5YZFDi0uIpfmolKQVGuKzbPpKPCyLAnt0iMGMi4gmzWV2i92WvcfvIzrJhMAofFQbbdxdiCdF7cdOS4zsexYATb1YREhSK1GTRFIoSwApcAf9EX/Q8wDs3tVQ38d7L1pJSPSynnSCnnFBQUnBBZu8MT9ABQ3Vbd49jctv2EMdNo0i7owZAeOM+04Q2GMQkT84rmUenfjCNNIIQgbD6CLVoSq7Lri5gJkEZOuo3iLM0VlRhsh7hrq2M2lOG6mjQ8l321rTS2Ban1BCh0aUH4qyddzcqylVx+egmfVDRyuNF7XOektxhJBXZlkSgUKc1g/oM/B2ySUtYASClrpJQRKWUUbcLjmYMoW69pDbYCUNVa1ePYwkAFddZSLDYtwu4Pahf8oiwHrXr21qIRiwjIFmwZh4nKKAFxBEukOLaNLVnn8ZvIpWQ50ijO1hRApxiJYZF0uEAbFsm0Ii1Qv76ikRq3n2GZ2ryQeUXzWD5uOStOLwHgpU8H1ipRFolCcXIwmIrkGhLcWkKIxObflwHbTrhEx4FhkRxtO4o0UrC6oDhSSaNzDFaLpjQCenvb4mw7bQHt7vyc0nMQWDClb6XSU0lUBBHB+KnZbpnKn6xfwGwSFGdrFkZHF1ambpF0tFSMGMmk4bm47BYeenUXR5p9FGS2L4lSmuNk7phcXvr0SI/H1BdiMRKV/qtQpDSD8g8WQjiBC4EXExb/PCGd+Dzg9sGQ7VgxFIk/4qfR39jluFA4wjDZQMBZhDUtDAha/QKzSTAs005bUFMu6Wnp5IhphB1b2NOkVaGJBuKKpMkbjJVOmViYSbrVHHNlGXTl2jIskgyrg99fdwZ1rQE8/jCFrs4lUa6YVcqB+jY+Pdx8LKfjmIhlbakJiQpFSjMoremklF4gr8OyrwyGLH3FHYr3Wq9uqybPkZd0XEtzI/kigMwowuStwyS1yYguu4UMm4W2QDjWITErOotGczkv73sZEAT9w2LbafaGYkH2q88YwZIphZ0uxF0G23WLxGq2csboXJ69aR53rNnMnFG5neT93PTh/OCv2/hbeRWzRibNezhuHvjHDiYXuTCaKyqLpGdCoRCVlZX4/f7BFkWRQtjtdkpLS0lL675aRV9RPU77iGGRgBYnmZY/Lfm42kPkAyKrmGBwPdbICFp8IVyONNJtFqJSu0N3WM3YgtPBauKdynfIMA2nNRhXCI1tQYqyNFeUxWximKtzpd7MWPpv8hiJUStranEW/779nKTyZtrTmD0qh40Hm3p5JnrPs+sPc+boXC6conVYVOm/PVNZWUlmZiajR4/uc3tjxamBlJKGhgYqKysZM2bMgO5L/YP7SKIi6S5zy9dwGIBIVj4+cQhTcAxuX4gsRxrpNk1RGAH3UMhBJpMAyEkbFSslAtDsDcYaXnVFbEJiF/NIDEXSE2UjstlZ7Y5lV/UH3mAYjz9MVYs/nrWlgu094vf7ycvLU0pE0WuEEOTl5Z0QK1Ypkj7iCXqwCAvpaendZm6FmrRZ6IetPkAS8Y7WLBJ7Gul6UNyrx0n8oQgFpjkAFFhHE4xEiUS1oHeTN0SOs3szNTPm2mpvcC4oXsCK8SvIsfXOVVU2IptwVLLtSP/1MKt1azPpq1t88awtZZH0CqVEFMfKifrNqH9wH/EEPWRaMylKL6KqrWtFIt3aZ/sjRwGBz1OC2x/WLRLtgm9YJL5QhJK0M5mYM5GxGZpCCYQj+EMRfKFIj6VLurJIJuRM4Mdn/RizqXcWQNnIbADK+zHgXqPX9mr2hmjyasUjVfqvQpHaKEXSR9xBN5nWTIoziqlu7dq1ZWo9SpPMYLdnJ7mWUbT5rTR7g7gclphry0gB9gUjZNlyeOGSFxiVfhqgxU+avEEg3vCqK+LB9r59vcMy7ZRkO/o1c6tGr+0FUFHfRppZy1xTKBSpi1IkfaS3FonVe5RqkcuWui2UOCYDUN8ajAXbgVgKsC8U78NuPPtDEZratDv4nlxbuU4rZpPoMZbSG8pGZFN+qP8UiVFtGKCioU1ZI4pesWjRIjZs2ADARRddRHPzwKWld6SiooJp05In0fSFn/70p/2+zcFCKZI+YiiS4oxiPEFPbKZ7R5yBWrbZs/GGvYzJjP8oXfa0mCuqLRCPkRhuKSOF1xeK0KxbJD0piJx0Ky9/8ywuLSvudlxvOH1kNkeafbEGWH2lNtEiaWhT5VFOAcLhcL9u75///CfZ2dn9us3BQCkSRYzWYKumSNK1i3ZXVokrVM/mdC1b6rTsGbHliTGStkCYUCRKKCJjisS4Y/eHIjTqiiS3F+Xdp5dm9cvdftmI/o2T1Lj9sfRlfyiqLJIUoaKigsmTJ3PjjTcydepUlixZgs/no7y8nHnz5jFjxgwuu+wympq0dPFFixZxzz33cO655/KrX/2K6667jptvvpnzzjuPsWPH8s477/C1r32NyZMnc91118X2c/PNNzNnzhymTp3Kj370o6SyjB49mvr6eh577DHKysooKytjzJgxnHfeeQC89tprzJ8/n1mzZnHVVVfR2pr85g7g/vvv54wzzmDatGncdNNNsUoOGzduZObMmcyfP59HH300Nn7u3Lls37499n7RokVs3LiRTz75hAULFnD66aezYMECdu/eDcBTTz3F5ZdfzrJly5gwYQLf/e53Abjrrrvw+XyUlZXx5S9/GYAVK1Ywe/Zspk6dyuOPPx7bx+9//3smTpzIokWLuPHGG1m5ciUAdXV1XHHFFZxxxhmcccYZfPDB4PUaVIqkj3iCHlxWF0UZ2uzzpHGSSIiMaDMfpruZnj+d0sz4THWXI410azxGYlglhiVi3LFrMZLeubb6k2klWVhMgr9truKW/9vEikc/iHVfPB5q3H5G5DjJz9CUoZqMmDrs3buXW265he3bt5Odnc0LL7zAV7/6VR566CG2bNnC9OnTue+++2Ljm5ubeeedd/j2t78NQFNTE2+++SYPP/wwy5cv5/bbb2f79u1s3bqV8vJyAB544AE2bNjAli1beOedd9iypeu+ed/4xjcoLy9n/fr1lJaWcscdd1BfX89PfvIT1q5dy6ZNm5gzZw6/+MUvutzGypUrWb9+Pdu2bcPn8/HKK68AcP3117N69Wo++uijduOvvvpqnnvuOQCqq6upqqpi9uzZTJo0iXfffZdPP/2U+++/n3vuuSe2Tnl5OWvWrGHr1q2sWbOGw4cP8+CDD+JwOCgvL+d///d/AXjyySfZuHEjGzZsYPXq1TQ0NFBVVcWPf/xj1q1bx+uvv86uXbti273tttu4/fbbWb9+PS+88AI33HBDr77HgUBNSOwjnpCHjLQMive+A3RhkXiO8ondRr3Zxx2TriHDFFcEHS2SQ3rF3RG52pwPI0YSCEVobuuda6s/saeZmVSUyd83V5FmFoQikk8ONHL2hPyeV05CrTvA5GIXvlCE+tYgNlUeJWUYM2YMZWVlAMyePZvPPvuM5uZmzj33XACuvfZarrrqqtj4L37xi+3WX758OUIIpk+fTmFhIdOnTwdg6tSpVFRUUFZWxnPPPcfjjz9OOBymurqaHTt2MGPGDLrjtttu4/zzz2f58uW88sor7Nixg7POOguAYDDI/Pnzu1z3rbfe4uc//zler5fGxkamTp3KOeec0+64vvKVr/Dqq68C8IUvfIELL7yQ++67j+eeey52vC0tLVx77bXs3bsXIQShULyd9eLFi8nK0lpHTJkyhYMHDzJixIhOsqxevZqXXnoJgMOHD7N3716OHj3KueeeS26uVn3iqquuYs8erXTS2rVr2bFjR2x9t9uNx+MhMzOz2/M1EChF0gdCkRC+sI9MayZ5O/9FmpRJLRLpPsIzrgzScbJ09FL21vhin7nsFtLMJqwWE63BMBUNmiIZnZcOJATbw5prK91qxnqC7+LvXDaJndVuLi0r4Zyfv8XanTXHrUhq3H4WnTaMUDjK1iMtyiJJIWy2+ERWs9ncY8A7PT096fomk6ndtkwmE+FwmAMHDrBq1SrWr19PTk4O1113XY+T6Z566ikOHjzII488AmizuS+88EKeeeaZbtcDbZLnN7/5TTZs2MCIESO499578fv9sVJFySgpKSEvL48tW7awZs0afvvb3wLwgx/8gPPOO4+XXnqJiooKFi1a1Om4QTtvyWJGb7/9NmvXruWjjz7C6XSyaNGimCxdEY1G+eijj3A4HD0e60Cj/sV9wBPSZrVnWjMx+RooCkeSTkr8rHor7zgdzHbMx2q2kmlrb5EApFvNeAMRKurbABiVZ1gkcddWszc04O1vk7FwQgE3nTOOQpeds8bn88aumuOqCtwaCNMWjFDossUqFytFkrpkZWWRk5PDe++9B8Cf/vSn2F388eB2u0lPTycrK4uampqYFdAVGzduZNWqVfz5z3/GZNJ+R/PmzeODDz5g3759AHi93tgdfEcMJZWfn09rayvPP/88ANnZ2WRlZfH+++8DxFxPBldffTU///nPaWlpiVlVLS0tlJRo7ReeeuqpXh1vWlpazHJpaWkhJycHp9PJrl27WLduHQBnnnkm77zzDk1NTYTDYV544YXY+kuWLIkpUCDmHhwM1L84CVJKvKGemzoZ5VEyrZnQVk9xOES153CncWsq38IEzC9cARCbNwJajERbphVurGhooyjLHk//TQi2J1b+HSwWTx7G4UYf+2q7DmB2hTEZsdBljwXcVeXf1Obpp5/mP//zP5kxYwbl5eX88Ic/PO5tzZw5k9NPP52pU6fyta99Leae6opHHnmExsZGzjvvPMrKyrjhhhsoKCjgqaee4pprrmHGjBnMmzevXVwhkezsbG688UamT5/OihUrOOOMM2Kf/eEPf+CWW25h/vz5ne74r7zySp599lm+8IUvxJZ997vf5e677+ass84iEuldSaGbbrqJGTNm8OUvf5lly5YRDoeZMWMGP/jBD5g3bx6gWUD33HMPc+fO5YILLmDKlCkxN9nq1avZsGEDM2bMYMqUKTz22GO92u+AIKVM2cfs2bPlQPBe5Xtyzp/myDpvXbfjttZtldOemibfPvSWlPcXyB/+erRc9MzZncat+OMC+bXfTJBv7jgqpZTSHwrLUXe+Ikfd+YoMhCJSSimXPvyOvPHp9fKyR9+XV//2o9i6R1t8ctSdr8g/r6uQ8366Vn7zzxv770CPg6pmrxx15yvyN2/tk22BkLz2yY/lr9/YIyORaI/rfrCvTo668xX5wb46+dfyI3LUna/IG59efwKkTn127Ngx2CIoBgmPxyOllDIUCsmLL75Yvvjii8e0frLfDrBB9uO1WMVIknCg5QD+iJ/9zfvJd3QdC3AHtRLymZghEqAobKM+0EwgEogVRoxEIxyKeJgeNJGboS2zWcykmQUWkykW70i3WWjTYyRLpxbG9mFYJLuPeqhu8TNvXPIy9SeKoiwHU4tdvLmrhqMtPt7eXcfbu+vYUtnCf39hZqzycDKMOluFLjtWs3bcyiJRKLrn3nvvZe3atfj9fpYsWcKKFSsGW6ROKEWSBENBHPYc5syirjv+xlxbYc3PWaIH0apbqxmdNRqASs9hgkKSHXS2m/+RYbO0m0PhtJo50uyjsS0YC7RDvKDhW7trAVgwyIoEYPHkQn795l7WVzRx3YLRjMx18sA/d/K5X73HqqtmMm9schkTXVv22DwZ5V1VDDyXXXYZBw4caLfsoYceYunSpYMkUe9ZtWrVYIvQI0qRJMEdiCuS7ogrEi0tt0hvHVvVVhVTJPs2PaFtyze1nSJJt1na9VrPsFk4EAu0JygSiwkh4HCjj0KXjbH57TNhBoMLJg9j9Rt7GZ3n5M5lk3BYzcwckcUdz23m6sfXceviCdxx4cRO69W4Azj1jo52iwmTUJV/FScGI61WMTAM2r9YCFGht9YtF0Js0JflCiFeF0Ls1Z/7tzVfL0m0SLrDUCSuoBaYL9YtkljmVt1u9m39PwBeDV3WSXEYGVugKRYjEWpMgrIQQsTu2uePHRr9KKYVZ3HzonE8+uVZsYmTs0fl8uptC7lo+nB+/eZeGlo7l1Sp9fgpdMWbcn11/mgWTRzWaZxCoUgtBvt28DwpZZmUco7+/i7gDSnlBOAN/f0J51gUiVmYcfi08cPCUcwITZGE/PD819lns5FOHjnOrHZKYMmUQhZPjsdC0hOUzEh9MqKB4QZaMO745m70NyaT4M5lk5hanNVuudNq4eZzxyMlvLW7Lrb84/0N3L6mnNd31MSytQDuvWQqF0wpRKFQpDZDzbV1KbBIf/008DZw54kWwnBtVbZWdjvOE/SQnpZBVXUlJYAleySFWLROiW//FGq2sm/KmaR5h+PqMP/jjiWntXtvzG4vyrJ36rWuBdxDzB8C8ZGemFbiotBl442dNVw5u5SNBxv54uPrcNktXD6rlK+dNXqwRVQoFP3MYFokEnhNCLFRCHGTvqxQSlkNoD8Pit+jJah1BPQEPbQEuu4O6Al5iITtvLF+G9KaCVkjKJa6RbL7X4TGXUCFvx5ChT0WWjQUiTERMRGH1UxpjiNWNmUoI4Rg8eRC3t1TRyAc4Zdr95KXbuXDuxfzs8unM6HwxJdvUPQds9kcK5BYVlZGRUVFl2MHquy6YugymBbJWVLKKiHEMOB1IUTyWUMd0JXOTQAjR44cEMHcATe59lwa/Y0c9hwmy5aVdJwn6EFG7bhkCxFnHpaMAoqbK/mktQqaD3Jw9FzCDXto8xRQOMKedBsGRin5xIwtg7PG5zHc1f36Q4kLJg/j/z4+xG/f2c97e+u563OTYsenSE2MAoMKRTIG7d8tpazSn2uFEC8BZwI1QogiKWW1EKIIqE2y3uPA4wBz5sw59jodPcuFO+hmXtE83jvyHoc9h5mWn/zuyhP0EAnZyaOagDUXS3oBRTVear21hMJ+9tm1GbHNLbmMmtm9NWEE4kcnycr6yYrpfTyqE8uCcfnY00w8vHYPOc40vjJv1GCLdFJx39+3s6PK3a/bnFLs4kfLpx7TOhUVFXzlK1+hrU3LNnzkkUdYsGBBuzHbt2/n+uuvJxgMEo1GeeGFF5gwYQJ//vOfWb16NcFgkLlz5/Kb3/wGs1nNKUpVBsW1JYRIF0JkGq+BJcA24G/Atfqwa4G/nmjZ/BE/oWiIKXlTgG4C7oc+xhNwEwhayRMeWs3ZkF5Aib+NKFFqLGb2iQgmTESDBYzqIW03bpEMffdVT9jTzJw9vgAp4YaFY2NuO0XqYvTOKCsr47LLLgNg2LBhvP7662zatIk1a9Zw6623dlrvscce47bbbqO8vJwNGzZQWlrKzp07WbNmDR988AHl5eWYzeZO9awUqcVg/cMLgZf0LCYL8H9Syn8JIdYDzwkhvg4cAq7qZhsDghFoH+YcRr4jn0pP+4D7vqZ97Dn0Hhe99C3cE6YQDE0iT7TQJLIoTM+nyJiUaLGwN9RCnr2EFpnGqB7iG1OLs5hc5GLWyEHJeO53rjlzBHUeP1+dr6yR/uZYLYf+IJlrKxQKsXLlypgySFYccf78+TzwwANUVlZy+eWXM2HCBN544w02btwYq23l8/kYNkylgacyg6JIpJT7gZlJljcAi0+8RHGM1F+XzcWIjFIOf/ZvGHUJlGoZyo+WP8rbh99kCeAJeyFiJxcPe6OZkF5AsT4p8S+ZmbxTu4HJGYvZT/IgeiIj85y8etvCgTy0E8riye3TmxUnHw8//DCFhYVs3ryZaDSK3d45jvelL32JuXPn8o9//IOlS5fyxBNPIKXk2muv5Wc/+9kgSK0YCAZ7HsmQI6ZIrC6GW1wcDrmJ7Hkd0OpmfXz0Y8IyyuE0Cz6iWKNm0kSEqlAGpBcwPBxGSMmrGU4m5kxktLgGl91yQptRKRQngpaWFoqKijCZTPzpT39KWvV2//79jB07lltvvZVLLrmELVu2sHjxYp5//nlqa7UQaGNjIwcPHjzR4iv6EaVIOmC4trKsWZhbTdSazVRVfQbAzsadsdns26yaYsiKan+ew4F0SC/AChSHIxRj5jcX/IYjTdF2JU8UipOFb37zmzz99NPMmzePPXv2dGpkBbBmzRqmTZtGWVkZu3bt4qtf/SpTpkzhJz/5CUuWLGHGjBlceOGFVFcnaVGtSBlUFLQDiRZJacCGFIL1bfsZAayrXhcbtz1Ta305SdYDsN/rgHRt5vmva+rInno5+Y58DjVsZWpJ8vRhhSJVaG3t3H9mwoQJ7XqqG66q0aNHs23bNgDuvvtu7r777k7rfvGLX+zUileRuiiLpAOJMZLJbSZyIhHeogGAdVXrmJg5mvRolG1ZBQCcadKyuvb7HPhN6WC2MSEUoiBvEuFIlMomX4+BdoVCoUhllCLpgDvoRiDISMvA1lbDslYvH1kj1Pvq2VS7ifmOIkaFQuwMawpnRkTz7TZIF0fdAUjXFAw5o6lq9hOOyh4D7QqFQpHKKEXSAXfATYY1A7PJjMN/lIvb2giYBD//+GeEoiHmhWBUOEowqvUgyYloz01kUtXii7m3yBnNwUZtotbIXBUjUSgUJy9KkXSgJdiCy+oCwBWsY3ogyMhQiFcPvobFZGFWUxWjbLmx8a5olIgtixAWqpr9cYskdwwHG7Ty8soiUSgUJzNKkXTAHXDHFElupI4D0eF8vlVTCDMLZuKs3srIrDGx8ZnRKEJXHlXNPsgsBEcOOHI41OjFajGlVJ0shUKhOFaUIumAO+jGZXMRCfrIxc0+6yQ+39qGAM7KmQL+ZkYP0+dSSoFTSkwZBeRn2Khq9uGdezsvTXyIxrYgBxvaGJHjwGQa/GZUCoVCMVCo9N8OuINuhjmH0VJ7iFzAk1/GqKNv85DjAs6zlQAwcvQiOPAMMmonas/BlJ5PSbadqhY/f96Tzk8/zmB65Se0BsLtuh0qFKlIQ0MDixdrBSeOHj2K2WymoECzwj/55BOsVjXZ9lRHKZIOGK4tT00FuUD2iCm0VDsZ6W7DfvAjsGeRVToXu8mFN2ghePGjWHJLKHojwt5aD8+u91KS7WBntZtwVHLuxILBPiSFok/k5eXF6mzde++9ZGRk8J3vfKfdGCklUkpMJuXkOBVRiiQBo4S8y+bCV6/NDykoGUsteZg81dB2CEadDSYz6WI4ftmGc9rnASjO3sG/th8F4L+unIHVYuJba8o5bbhq5KToZ169C45u7d9tDp8On3vwmFbZt28fK1as4Oyzz+bjjz/m5ZdfZubMmTQ3NwPw7LPPsnbtWp544glqamq4+eabOXQ4iFixAAATLklEQVToECaTidWrVzNv3rz+PQbFoKEUSQJGCXmX1UW4Wav66xo2kgZrAWNbt0GkEeZ+A4DCyBUQjM/2Lc7WAuqZdgsXzyjGYTUza2ROux7lCsXJxo4dO/jDH/7AY489RlivfJ2MW2+9le9+97vMmzePiooKLr744tjsd0XqoxSJQcCDu/JjALIsTnBX4ZZO8vPyOOIYTo57EwD/tWcYL771BrWebBZNnBBbvSRba2K1oqwk1nM9FVrjKlKQY7QcBpJx48bFysF3x9q1a9m9e3fsfVNTEz6fD4fDMZDiKU4QSpEAeGpgdRluQlBahGv/u1jaqqkhl/FWM9GMInCD15LFb3ZY+dz0bIqyHFwyszi2iZkjspk0PJNrF4wevONQKE4wiYUaTSYTUsablvr9/thrKaUKzJ/EqMgYwMH3IeTFfdZKAFy7/02O9wAN5gKEEKTlaNlabwdO4wtzRvGbL8/mBxdPYeaI7NgmirMd/Otb5zB+WMagHIJCMdiYTCZycnLYu3cv0WiUl156KfbZBRdcwKOPPhp7r/q/n1woRQJw+BOwOHCPPhsAl99DYfAwHqvWtS2jQOvydyBjFvddeuK70ykUqcJDDz3EsmXLWLx4MaWlpbHljz76KB988AEzZsxgypQp/O53vxtEKRX9jXJtgaZISmbjDmu1sVwlc+HA+/gdwwEYM+t8Pi6/iIuvuAV7mnkwJVUoBpV777039nr8+PGdLIuuysMXFBTw/PPPD7R4ikHihFskQogRQoi3hBA7hRDbhRC36cvvFUIcEUKU64+LTohAQS/y6BZa8k+n3qf1Fsk561sA+DNGAJDuymXut55h1IiRJ0QkhUKhSCUGwyIJA9+WUm4SQmQCG4UQr+ufPSylXHVCpan6FBENc8c6GxbrVkoySrCNWcwlgR9zwYhBbR+vUCgUKcEJt0iklNVSyk36aw+wEyg50XIYtH32IQCbIuPZVreNKXlTaGgNskWOIy9LBc4VCoWiJwY12C6EGA2cDnysL1ophNgihHhSCJHTxTo3CSE2CCE21NXV9VmGQ5vfZr8swpmbiTtSw5S8KdS3BgDIz7D1efsKhUJxsjNoikQIkQG8AHxLSukG/gcYB5QB1cB/J1tPSvm4lHKOlHKOUTjuePms1sOwls205J/O2VO1nPdS5wR2HfUAUJCpFIlCoVD0xKAoEiFEGpoS+V8p5YsAUsoaKWVEShkFfgecOdBylJdvJE94GF12HlnZNQDsOpjFg6/uZHKRi+klWQMtgkKhUKQ8g5G1JYDfAzullL9IWF6UMOwyYMAL8YT2vwdAzqRF1Ic+Q4Tz+NXrR3D7wvzyi2WkmdU0G4WioaGBsrIyysrKGD58OCUlJbH3wWDwhMlx+PDhWGrxpk2b+Ne//hX77Pvf/z6//OUve9xGaWkp06dPp6ysjOnTp/P3v/+9X2VMlON73/seb731Vr9uf6gyGFlbZwFfAbYKIYwk9HuAa4QQZYAEKoD/N9CC5NZvoMWcQ1b+BHY07GC4fTxuCd9ddpqq2qtQ6AyVMvIjRoxgzZo1gKZItm3bxrJly455O++99x7Z2dls376dSy65hOXLl/e3qAA88MADA7LdocgJVyRSyveBZC0D/3ki5WhuCzAptJV/F03iXG8tR1qPcN2kFVxeOomvnTWm5w0oFIPEQ588xK7GXf26zUm5k7jzzDuPaZ3+LiO/ZMkSfvnLXzJlyhSmT5/ONddcwz333MPdd9/Naaedxtlnn82VV17JRx99xP3334/P5+Ptt9/m+9//PgBbt27l3HPP5fDhw3z729/mlltu6VZ+t9tNTk48p2f58uVUVVXh9/u5/fbbueGGGwiHw1x//fWUl5cjpeSmm27i1ltvZe/evaxcuZL6+nrS09N54oknmDhxYrvt/8d//AdXXnklK1asoLS0lBtuuIG//vWvRCIRnn/+eSZOnEhraysrV65kx44dhEIh7r///gFTbAPJKTuzfc/u7Rxy+fmx4wi//OtlAJw1sox5ReMGWTKFInXozzLy55xzDu+99x5FRUXY7Xbef/99AN5//31uuOGGWEFIh8PBD3/4Q7Zt2xZzI5WXl7Nnzx7eeOMNmpubmTx5Mt/4xjcwmztXoli4cCHRaJQDBw7w4osvxpY//fTT5Obm4vV6mTNnDldccQV79uyhvr6erVu1/i+Gkrzpppt44oknGDduHB988AErV67ktdde6/ZcFRYW8umnn7J69Wp+8Ytf8Nhjj3H//fezbNkynnrqKZqampg7dy4XXnghdntqtZ84ZRVJ8863eTYzk1JHIXZbJpHWCFPypgy2WApFjxyr5TCQ9GcZ+YULF/L4449TVFTEpZdeyj/+8Q+8Xi9Hjhxh3Lhx7Nu3r9t9XHzxxVitVoYNG0Zubi51dXUMHz680zjDtbVnzx6WLl3K9u3bcTqdPPzww/ztb38DoLKyks8++4zx48eze/dubrvtNi666CKWLFlCc3Mz69at44orrohtszslanD55ZcDMHv2bP75T80B89prr/Hqq6/y4INaawC/38+hQ4c6WTdDnVNWkVTXvcnuHCs/nHETl028nLZQGy6ra7DFUihSiv4sIz937lxuuOEGiouLWb58OZWVlfzud7/jzDN7l8Bps8XT9c1mc48X94kTJ5Kbm8uuXbtobGzk3XffZd26dTgcDs4++2z8fj95eXls2bKFV199ldWrV/PCCy/w0EMPkZ+ff8wVjA35EmWTUvLyyy8zblxqe0JOybQkKSUb0vbjiAo+P+5iLCYLWTaV6qtQ9IW+lpG32+0UFhby8ssvM3fuXBYuXMiqVatYuHBhp7GZmZl4PJ4+yXv06FEOHTrEyJEjaWlpITc3F4fDwfbt21m/fj0AdXV1SCm56qqruO+++9i0aRM5OTkUFRXFji8ajbJ58+bjkmHp0qWsXr069v7TTz/t0zENFqekItn52UbeTzdxlmU8zjTVxVCh6C/6WkZ+4cKFFBUVYbPZWLhwIZWVlUkVyfnnn8/mzZs5/fTTj7mq8MKFCykrK2Px4sWsWrWK/Px8Pv/5z+P1epk5cyb3338/c+fOBbSU43POOYeysjJuvPFGfvrTnwJaIsFjjz3GzJkzmTp1Kq+88soxyWDwox/9CK/Xy/Tp05k6dWq76sqphEg0RVONOXPmyA0bNhzzeq9//By/3vIAd8z8EYvOvHwAJFMo+pedO3cyefLkwRZDkYIk++0IITZKKef01z5OyRjJhXO/wIVzvzDYYigUCsVJwSnp2lIoFApF/6EUiUKRIqSyG1oxOJyo34xSJApFCmC322loaFDKRNFrpJQ0NDSckMmNp2SMRKFINUpLS6msrKQ/evAoTh3sdnu77LmBQikShSIFSEtLY8wYVQNOMTRRri2FQqFQ9AmlSBQKhULRJ5QiUSgUCkWfSOmZ7UKIOuDgcayaD9T3szgDTSrKDKkpdyrKDKkpdyrKDKkv9ygpZUF/bTSlFcnxIoTY0J/lAU4EqSgzpKbcqSgzpKbcqSgzKLk7olxbCoVCoegTSpEoFAqFok+cqork8cEW4DhIRZkhNeVORZkhNeVORZlByd2OUzJGolAoFIr+41S1SBQKhULRTyhFolAoFIo+cUopEiHEMiHEbiHEPiHEXUNAngohxFYhRLkQYoO+LFcI8boQYq/+nKMvF0KI1brsW4QQsxK2c60+fq8Q4toBkPNJIUStEGJbwrJ+k1MIMVs/D/v0dcUAyn2vEOKIfs7LhRAXJXx2ty7DbiHE0oTlSX83QogxQoiP9eNZI4Sw9oPMI4QQbwkhdgohtgshbtOXD9nz3Y3MQ/1c24UQnwghNuty39fdvoQQNv39Pv3z0cd7PAMk91NCiAMJ57tMXz7wvxEp5SnxAMzAZ8BYwApsBqYMskwVQH6HZT8H7tJf3wU8pL++CHgVEMA84GN9eS6wX3/O0V/n9LOc5wCzgG0DISfwCTBfX+dV4HMDKPe9wHeSjJ2i/yZswBj9t2Lu7ncDPAdcrb9+DLi5H2QuAmbprzOBPbpsQ/Z8dyPzUD/XAsjQX6cBH+vnMOm+gG8Cj+mvrwbWHO/xDJDcTwFXJhk/4L+RU8kiORPYJ6XcL6UMAs8Clw6yTMm4FHhaf/00sCJh+R+lxjogWwhRBCwFXpdSNkopm4DXgWX9KZCU8l2gcSDk1D9zSSk/ktov+I8J2xoIubviUuBZKWVASnkA2If2m0n6u9Hv0M4HntfXTzwHfZG5Wkq5SX/tAXYCJQzh892NzF0xVM61lFK26m/T9IfsZl+J38HzwGJdtmM6ngGUuysG/DdyKimSEuBwwvtKuv+xnwgk8JoQYqMQ4iZ9WaGUshq0PygwTF/elfyDdVz9JWeJ/rrj8oFkpW7iP2m4iHqQL9nyPKBZShnusLzf0F0np6PdcabE+e4gMwzxcy2EMAshyoFatAvpZ93sKyaf/nmLLtsJ/292lFtKaZzvB/Tz/bAQwtZR7l7Kd8y/kVNJkSTz8Q127vNZUspZwOeAW4QQ53Qztiv5h9pxHaucJ1r+/wHGAWVANfDf+vIhJbcQIgN4AfiWlNLd3dAu5DjhcieRecifayllREpZBpSiWRCTu9nXkJVbCDENuBuYBJyB5q66Ux8+4HKfSoqkEhiR8L4UqBokWQCQUlbpz7XAS2g/5BrdtER/rtWHdyX/YB1Xf8lZqb/uuHxAkFLW6H/CKPA7tHN+PHLXo7kILB2W9xkhRBraBfl/pZQv6ouH9PlOJnMqnGsDKWUz8DZaDKGrfcXk0z/PQnOdDtp/M0HuZbqLUUopA8AfOP7zfey/kd4Ed06GB1o3yP1owTAj8DV1EOVJBzITXn+IFtv4L9oHVX+uv/487QNmn8h4wOwAWrAsR3+dOwDyjqZ90Lrf5ATW62ONwN5FAyh3UcLr29F82wBTaR8w3Y8WLO3ydwP8hfZB2W/2g7wCzSf9yw7Lh+z57kbmoX6uC4Bs/bUDeA+4uKt9AbfQPtj+3PEezwDJXZTwffwSePBE/Ub69WIz1B9o2Qt70Pyg3xtkWcbqP6zNwHZDHjSf6xvAXv3Z+GIF8Kgu+1ZgTsK2voYW4NsHXD8Asj6D5poIod2tfL0/5QTmANv0dR5Br7gwQHL/SZdrC/A32l/svqfLsJuELJWufjf6d/iJfjx/AWz9IPPZaG6ELUC5/rhoKJ/vbmQe6ud6BvCpLt824Ifd7Quw6+/36Z+PPd7jGSC539TP9zbgz8Qzuwb8N6JKpCgUCoWiT5xKMRKFQqFQDABKkSgUCoWiTyhFolAoFIo+oRSJQqFQKPqEUiQKhUKh6BNKkSgUCoWiTyhFokhphBDZQohv9jBmtBDiS73Y1miRUHI+yedlHUqhX9JfpcG72N8KIcSUgdq+QtFfKEWiSHWy0cp7d8dooEdF0gvK0CaYASCl/JuU8sF+2G5XrEArUa5QDGnUhERFSiOEMEpz70ar3gpaEUwJ/ERKuUYIsQ6tGN8BtDLgL6HNuk7Xx6+UUn6oV659RUo5Lcl+rGizfx3AEeBn+us5UsqVQoinAB9a0bxRwPXAtWg9HT6WUl6nb2cJcB9aOY3P0GYTtwohHgQuAcLAa8CLwCtoFWZbgCt0UR5FK5HhBW6UUu7S9+1HK9VRCNwhpXxFCDEVreaSFe2m8Qop5d5jO8MKRS/ojyn76qEeg/UgoZYW2sX2dbQ6R4XAIbSmS4vQFISxjhOw668nABs6bquLfV0HPJLsPVpToWfRylFcCriB6WgX8I1o1kw+8C6Qrq9zJ/BDtJpHu4nf2GUnbPPKhP29AUzQX88F3kwY9y99XxPQysHYgV8DX9bHWAHHYH9f6nFyPowKlwrFycDZwDNSyghatdx30EpqdyzDngY8orcijQAT+2n/f5dSSiHEVqBGSrkVQAixHU1JlaK5qj7QO5dagY90+fzAE0KIf6BZIu3QS7QvAP6S0PXUljDkOalV2d0rhNiPZhl9BHxPCFEKvCiVNaIYIJQiUZxM9Lb3+O1ADTAT7S7e30/7D+jP0YTXxnsLmtJ6XUp5TccVhRBnAovRqsquROvSl4gJreFSWRf77uijllLK/xNCfIxW/fXfQogbpJRvHssBKRS9QQXbFamOB61POGhuoy/q3eMK0Hq2f9JhDGh9JKr1O/ivoLnCjnVfx8M64CwhxHgAIYRTCDFRtzaypJT/BL6F5gZrtz+pNYo6IIS4Sl9XCCFmJmz7KiGESQgxDq167W4hxFhgv5RyNVr13Rl9kF2h6BKlSBQpjZSyAc1VtA0tsL0FrTT/m8B3pZRH9WVhIcRmIcTtwG+Aa/Ug/ESgrZe7ewuYIoQoF0J88ThkrUOLqzwjhNiCplgmoSmLV/Rl76BZTKDFXP5TCPGpriC+DHxdCGG0Hkjs/71bX/dV4BtSSj/wRWCb3pJ1ElrPEIWi31FZWwpFiqNnbb0ipXx+sGVRnJooi0ShUCgUfUJZJApFB4QQS4GHOiw+IKW8bDDkUSiGOkqRKBQKhaJPKNeWQqFQKPqEUiQKhUKh6BNKkSgUCoWiTyhFolAoFIo+8f8Dyrq18AIZH0YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "pg_result_no_na[\"normalize_advantage\"] = False\n",
    "pg_result_na[\"normalize_advantage\"] = True\n",
    "pgwb_result_wb[\"normalize_advantage\"] = \"True with Baseline\"\n",
    "pg_result = pd.concat([pg_result_no_na, pg_result_na, pgwb_result_wb])\n",
    "ax = sns.lineplot(\n",
    "    x=\"total_timesteps\", \n",
    "    y=\"performance\", \n",
    "    data=pg_result, hue=\"normalize_advantage\",\n",
    ")\n",
    "ax.set_title(\"Policy Gradient\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average episode reward for your Policy Gradient agent in CartPole-v0:  183.0\n"
     ]
    }
   ],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "# You should see a pop up window which display the movement of the cart and pole.\n",
    "print(\"Average episode reward for your Policy Gradient agent in CartPole-v0: \",\n",
    "      pg_trainer_wb.evaluate(1, render=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 4: Actor-critic Method\n",
    "\n",
    "(30 / 100 points totally)\n",
    "\n",
    "### Section 4.1: Implement ActorCriticTrainer\n",
    "\n",
    "(20 / 100 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As point out in the textbook Page 331, the difference of Actor-critic methods and policy gradient with baseline method is whether the critic (baseline) participates in the bootstrapping. In `PolicyGradientWithBaselineTrainer`, we update the baseline network by fitting the ground-truth Q values. However, in actor-critic trainer presented below, we update the critic via fitting a bootstrapped target: \n",
    "\n",
    "$$\\delta = r_t + \\gamma V(s_{t+1}) - V(s_t)$$\n",
    "\n",
    "wherein $V$ is the values predicted by critic. In `ActorCriticTrainer`, in order to reuse the codes, we use `self.baseline` to represent the critic network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solve the TODOs and remove `pass`\n",
    "\n",
    "ac_default_config = merge_config(dict(\n",
    "    normalize_advantage=True,\n",
    "    num_critic_updates=10,\n",
    "    num_critic_update_steps=10\n",
    "), pg_with_baseline_default_config)\n",
    "\n",
    "class ActorCriticTrainer(PolicyGradientWithBaselineTrainer):\n",
    "    def __init__(self, config=None):\n",
    "        config = check_and_merge_config(config or {}, ac_default_config)\n",
    "        super().__init__(config)\n",
    "        \n",
    "    def process_samples(self, samples):\n",
    "        \"\"\"\n",
    "        In this function, we will compute the target of value function\n",
    "        and also the advantage used for updating policy.\n",
    "        \n",
    "        First, we need to prepare data to compute the values.\n",
    "        \n",
    "        Consider you only collect two trajectories in samples.\n",
    "        Then the data structure should be:\n",
    "            samples[\"obs\"] = [\n",
    "                [\n",
    "                    obs_t1_0,  <<= t=0\n",
    "                    obs_t1_1,  <<= t=1\n",
    "                    obs_t1_n1-1, <<= t=n1-1 \n",
    "                ],  (the traj t1 has length = n1)\n",
    "                [\n",
    "                    obs_t2_0,\n",
    "                    ...\n",
    "                    obs_t2_n2-1 \n",
    "                ]   (the traj t2 has length = n2)\n",
    "            ]\n",
    "        the whole training batch size is N = (n1 + n2).\n",
    "        Then what you should do is to take store\n",
    "        [\n",
    "            obs_t1_1, ..., obs_t1_n1-1, ZERO, \n",
    "            obs_t2_1, ... obs_t2_n2-1, ZERO\n",
    "        ]\n",
    "        into `next_obs` variable, wherein ZERO is a zero array has\n",
    "        same shape as any other observation.\n",
    "        \n",
    "        Besides, you also need to prepare a boolean array `masks`\n",
    "        who has length N. It should be like:\n",
    "        [\n",
    "            False(0), ..., False(n1-2), True(n1-1),\n",
    "            False(0), ..., False(n2-2), True(n2-1)\n",
    "        ].\n",
    "        It represents whether a transition is terminal transition\n",
    "        and whether a observation in next_obs should be masked.\n",
    "        We will used it when computing the bootstrapped values:\n",
    "        \n",
    "            value_t = reward_t + gamma * value_t+1 * (1 - masks)\n",
    "            \n",
    "        So that we will not consider the impact of value_t+1\n",
    "        in terminal state.\n",
    "        \n",
    "        Then, we need to compute the values and advantage based on \n",
    "        the data collected and form a processed_samples for the \n",
    "        consecetive .\n",
    "        \"\"\"\n",
    "        # Define the sum of all trajectory length as N (train batch size)\n",
    "        N = sum([len(traj) for traj in samples[\"obs\"]])\n",
    "        n = len(samples[\"obs\"][0])  # the first traj length\n",
    "        \n",
    "        # [TODO] Create an array named next_obs.\n",
    "        # In each trajectory, take the observations from second\n",
    "        # transitions to the end. Fill an extra all zero observation\n",
    "        # at the end of trajectory to align next_obs with others.\n",
    "        next_obs = []\n",
    "        for obs_list in samples[\"obs\"]:\n",
    "            pass\n",
    "            \n",
    "        next_obs = np.array(next_obs)\n",
    "        \n",
    "        assert next_obs.shape == (N, self.obs_dim)\n",
    "        assert np.all(next_obs[n - 1] == 0.0)\n",
    "        \n",
    "        # [TODO] Scan all trajectories and create a boolean array `masks`.\n",
    "        # It should be N length, N is the sum of the trajectories lengths.\n",
    "        # You should loop over each trajectory and store n-1 False into masks\n",
    "        # while storing one True at the end of each episode. n is the length\n",
    "        # of current trajectory.\n",
    "        # Finally, you should transform the list to an float32 array\n",
    "        # with shape (N,)\n",
    "        masks = []\n",
    "        for obs_list in samples[\"obs\"]:\n",
    "            pass\n",
    "            \n",
    "        masks = np.array(masks, dtype=np.float32)\n",
    "        \n",
    "        assert masks.shape == (N,)\n",
    "        assert masks[n - 1] == True\n",
    "        \n",
    "        # flatten rewards\n",
    "        rewards = np.concatenate(samples[\"reward\"])\n",
    "        tensor_rewards = self.to_tensor(rewards)\n",
    "        \n",
    "        # flatten observations\n",
    "        obs = np.concatenate(samples[\"obs\"])\n",
    "        \n",
    "        # [TODO] Compute the bootstrapped values\n",
    "        # Hint: value_t = reward_t + gamma * value_t+1 * (1 - masks)\n",
    "        #  You need to ask self.baseline using next_obs to get \n",
    "        #  the values in next state, then can you compute the \n",
    "        #  value_t.\n",
    "        tensor_next_obs = self.to_tensor(next_obs)\n",
    "        values = None\n",
    "        pass\n",
    "        \n",
    "        assert isinstance(values, np.ndarray)\n",
    "        assert values.shape == (N,)\n",
    "        \n",
    "        # [TODO] Compute the baseline by feeding obs to critic\n",
    "        tensor_obs = self.to_tensor(obs)\n",
    "        baselines = None\n",
    "        pass\n",
    "        \n",
    "        assert isinstance(baselines, np.ndarray)\n",
    "        assert baselines.shape == (N,)\n",
    "        \n",
    "        # [TODO] Compute the advantages using values and baselines\n",
    "        advantages = None\n",
    "        pass\n",
    "        \n",
    "        if self.config[\"normalize_advantage\"]:\n",
    "            # [TODO] normalize the advantages\n",
    "            pass\n",
    "        \n",
    "        # We passed part of numpy array and part of torch tensor\n",
    "        # to the following modules.\n",
    "        processed_samples = samples\n",
    "        processed_samples[\"advantages\"] = advantages\n",
    "        processed_samples[\"tensor_next_obs\"] = tensor_next_obs\n",
    "        processed_samples[\"tensor_obs\"] = tensor_obs\n",
    "        processed_samples[\"tensor_rewards\"] = tensor_rewards\n",
    "        processed_samples[\"tensor_masks\"] = self.to_tensor(masks)\n",
    "        process_info = {\n",
    "            \"mean_baselines\": float(np.mean(baselines)),\n",
    "            \"advantages_std\": float(advantages.std())\n",
    "        }  # You can add value here so that it will be printed automatically.\n",
    "        return processed_samples, process_info\n",
    "        \n",
    "    def update_baseline(self, processed_samples):\n",
    "        # Use a bootstrapped target values and update the critic.\n",
    "        # Compute the target values r(s, a) + gamma * V(s') by calling\n",
    "        # the critic to compute V(s').\n",
    "        reward = processed_samples[\"tensor_rewards\"]\n",
    "        obs = processed_samples[\"tensor_obs\"]\n",
    "        next_obs = processed_samples[\"tensor_next_obs\"]\n",
    "        masks = processed_samples[\"tensor_masks\"]\n",
    "        \n",
    "        assert masks.shape == reward.shape\n",
    "        assert next_obs.shape == obs.shape\n",
    "        \n",
    "        losses = []\n",
    "        \n",
    "        for _ in range(self.config[\"num_critic_updates\"]):  # Train critic for 10 iteration\n",
    "            self.baseline.eval()\n",
    "            \n",
    "            # Predict the values of next state\n",
    "            values_next = self.baseline(next_obs).reshape(reward.shape)  # a tensor\n",
    "            \n",
    "            assert values_next.shape == reward.shape  # tensor shape should equal\n",
    "            \n",
    "            # [TODO] Compute the target of critic using reward, masks\n",
    "            # and values_next (don't forget gamma)\n",
    "            # Hint: the masks are all False (that is, zero) when the state is not\n",
    "            #  terminal state, and are True (one) when is. So we can use (1-masks)\n",
    "            #  as a multiplier to apply to values_next when computing the target.\n",
    "            target = None\n",
    "            pass\n",
    "            \n",
    "            target = target.detach()\n",
    "            self.baseline.train()\n",
    "\n",
    "            for _ in range(self.config[\"num_critic_update_steps\"]):\n",
    "                # [TODO] Uncomment the whole section\n",
    "                \"\"\"\n",
    "                baselines = self.baseline(obs).reshape(reward.shape)\n",
    "                loss = self.baseline_loss(input=baselines, target=target)\n",
    "                self.baseline_optimizer.zero_grad()\n",
    "                loss.backward()\n",
    "                torch.nn.utils.clip_grad_norm_(\n",
    "                    self.baseline.parameters(), self.config[\"clip_gradient\"]\n",
    "                )\n",
    "                self.baseline_optimizer.step()\n",
    "                losses.append(loss.item())\n",
    "                \"\"\"\n",
    "                pass\n",
    "            \n",
    "        self.baseline.eval()\n",
    "        return {\"critic_loss\": np.mean(losses)}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "ac_trainer, ac_result = run(ActorCriticTrainer, dict(\n",
    "    learning_rate=0.01,\n",
    "    max_episode_length=200,\n",
    "    train_batch_size=200,\n",
    "), 195.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "pg_result_na[\"label\"] = \"Pure PG\"\n",
    "pgwb_result_wb[\"label\"] = \"PG with Baseline\"\n",
    "ac_result[\"label\"] = \"Actor-critic\"\n",
    "pg_result = pd.concat([pg_result_na, pgwb_result_wb, ac_result])\n",
    "ax = sns.lineplot(\n",
    "    x=\"total_timesteps\", \n",
    "    y=\"performance\", \n",
    "    data=pg_result, hue=\"label\",\n",
    ")\n",
    "ax.set_title(\"Policy Gradient\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 4.2 -- Land on the Moon!\n",
    "\n",
    "(5 / 100 points)\n",
    "\n",
    "Let's try to land on the moon! In this section, we will try a harder environment: \"LunarLander-v2\". The agent needs to control the lander to land on a small interval of the moon marked by the flags. Due to the complex dynamic of rocket, the environment is hard to solve and it may take hours for you to train. \n",
    "\n",
    "The environment provides a 8-element vector as observation, which is different from the image observation provided by the following Pong environment in the next setion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell without modification\n",
    "e = gym.make(\"LunarLander-v2\"); e.reset()\n",
    "frames = []\n",
    "for _ in range(1000):\n",
    "    frames.append(e.render(\"rgb_array\"))\n",
    "    if e.step(e.action_space.sample())[2]: break\n",
    "e.close();animate(frames);del e"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "e = gym.make(\"LunarLander-v2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "e.observation_space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "ac_trainer_lander, ac_result_lander = run(ActorCriticTrainer, dict(\n",
    "    env_name=\"LunarLander-v2\",\n",
    "    max_iteration=500,\n",
    "    learning_rate=0.005,\n",
    "    max_episode_length=1000,\n",
    "    train_batch_size=10000,\n",
    "    normalize_advantage=True,\n",
    "    evaluate_interval=1,\n",
    "    evaluate_num_episodes=5,\n",
    "), 200)\n",
    "\n",
    "# Hint: 1. This would take hours to train on personal laptop.\n",
    "#       2. Episode reward should greater than 0 after 80 iterations.\n",
    "#       3. We are using purely on-policy algorithm, so we expect the performance \n",
    "#          to be highly unstable.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "# Plot the result to diagnose potential problem.\n",
    "sns.lineplot(\n",
    "    x=\"total_timesteps\", y=\"performance\", data=ac_result_lander\n",
    ").set_title(\"Episode Reward in LunarLander-v2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "# You should see a pop up window which display the movement of the cart and pole.\n",
    "print(\"Average episode reward for your Actor-critic agent in LunarLander-v2: \",\n",
    "      ac_trainer_lander.evaluate(1, render=True))\n",
    "print(\"One small step for agent, one giant leap for mankind.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 4.3: Even harder -- train Pong agents\n",
    "\n",
    "(5 / 100 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell without modification\n",
    "e = gym.make(\"PongDeterministic-v4\"); e.reset()\n",
    "frames = []\n",
    "for _ in range(1000):\n",
    "    frames.append(e.render(\"rgb_array\"))\n",
    "    if e.step(e.action_space.sample())[2]: break\n",
    "e.close();animate(frames);del e"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "In this section, we will train agents to play Pong game in Atari environment. The agent control the bat on the right with up or down action to make it hit the table tennis ball to left side. The agent need to beat a rule-based AI.\n",
    "\n",
    "The observation of the environment `PongDeterministic-v4` is an image of 210 X 160 X 3 pixels. We compress the image and flatten it to be a vector with 6400 elements. To do this, we provide you a wrapper class which wrap the actor-critic trainer to fit this environment. You need to implement the save and restore functions.\n",
    "\n",
    "In the assignment 4, we will trained state of the art Pong agents with various algorithms. Your agents will battle with other AI. Let's get familiar with Pong by finishing this section!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "def process_img(img):\n",
    "    \"\"\"In this function, we process the raw image array into a flat vector.\n",
    "    We assume the img has four dimension, with shape in [Batch size, Width, Height, Channel].\n",
    "    Therefore the raw image is a 4D matrix containing the intensity of colors.\n",
    "    \"\"\"\n",
    "    assert img.shape[-3:] == (210, 160, 3), img.shape\n",
    "    \n",
    "    # Add extra axis if input is a single image and make it looks like a batch.\n",
    "    single_img = False\n",
    "    if img.ndim == 3:\n",
    "        img = img[None, ...]\n",
    "        single_img = True\n",
    "        \n",
    "    # Crop the image\n",
    "    img = img[:, 35:195]\n",
    "    \n",
    "    # Shrink the image size by taking only 1/4 of the pixel\n",
    "    img = img[:, ::2, ::2, 0]\n",
    "    \n",
    "    # Erase background\n",
    "    img[img == 144] = 0\n",
    "    img[img == 109] = 0\n",
    "    \n",
    "    # Paint everything (paddles, ball) to 1\n",
    "    img[img != 0 ] = 1\n",
    "    \n",
    "    # Flatten image to vector\n",
    "    img = img.astype(np.float).reshape(-1, 6400)\n",
    "    \n",
    "    # Reverse single image\n",
    "    if single_img:\n",
    "        img = img[0, ...]\n",
    "    return img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solve the TODOs and remove `pass`\n",
    "\n",
    "def pong_wrapper(trainer):\n",
    "    assert inspect.isclass(trainer)\n",
    "    \n",
    "    class WrappedTrainer(trainer):\n",
    "        def __init__(self, *a, **k):\n",
    "            super().__init__(*a, **k)\n",
    "            self.obs_dim = 6400\n",
    "            self.build_model()\n",
    "            assert \"Pong\" in self.config[\"env_name\"]\n",
    "            \n",
    "        def compute_action(self, obs):\n",
    "            obs = process_img(obs)\n",
    "            return super().compute_action(obs)\n",
    "            \n",
    "        def process_samples(self, samples):\n",
    "            samples[\"obs\"] = [process_img(traj_obs) for traj_obs in samples[\"obs\"]]\n",
    "            return super().process_samples(samples)\n",
    "        \n",
    "        def train(self):\n",
    "            ret = super().train()\n",
    "            if self.iteration % self.config[\"evaluate_interval\"] == 0:\n",
    "                self.save(\"iter{}\".format(self.iteration))\n",
    "                print(\"Finished {} iterations training. \"\n",
    "                      \"Checkpoint is saved in checkpoints directory.\".format(self.iteration))\n",
    "            return ret\n",
    "        \n",
    "        def save(self, surfix=\"checkpoint\"):\n",
    "            import pickle\n",
    "            import os\n",
    "            os.makedirs(\"checkpoints\", exist_ok=True)\n",
    "            policy_file_name = \"checkpoints/pong-agent-{}-policy.pkl\".format(surfix)\n",
    "            with open(policy_file_name, \"wb\") as f:\n",
    "                # [TODO] Use function of pickle package and torch model to save \n",
    "                # the parameters of self.policy to file \"f\".\n",
    "                # Hint: a function called state_dict of torch model may help.\n",
    "                pass\n",
    "            \n",
    "            baseline_file_name = \"checkpoints/pong-agent-{}-baseline.pkl\".format(surfix)\n",
    "            with open(baseline_file_name, \"wb\") as f:\n",
    "                # [TODO] Use function of pickle package and torch model to save \n",
    "                # the parameters of self.baseline to file \"f\".\n",
    "                pass\n",
    "                    \n",
    "        def restore(self, surfix=\"checkpoint\"):\n",
    "            import pickle\n",
    "            policy_file_name = \"checkpoints/pong-agent-{}-policy.pkl\".format(surfix)\n",
    "            with open(policy_file_name, \"rb\") as f:\n",
    "                # [TODO] Use function of pickle package and torch model to restore \n",
    "                # the parameters of self.policy from file \"f\".\n",
    "                # Hint: a function called load_state_dict of torch model may help.\n",
    "                pass\n",
    "                \n",
    "            baseline_file_name = \"checkpoints/pong-agent-{}-baseline.pkl\".format(surfix)\n",
    "            with open(baseline_file_name, \"rb\") as f:\n",
    "                # [TODO] Use function of pickle package and torch model to restore \n",
    "                # the parameters of self.baseline from file \"f\".\n",
    "                pass\n",
    "        \n",
    "    return WrappedTrainer\n",
    "\n",
    "\n",
    "# Test save and restore\n",
    "pong_wrapper(ActorCriticTrainer)({\"env_name\": \"Pong-v0\"}).save(\"test\")\n",
    "pong_wrapper(ActorCriticTrainer)({\"env_name\": \"Pong-v0\"}).restore(\"test\")\n",
    "import os\n",
    "os.remove(\"checkpoints/pong-agent-test-baseline.pkl\")\n",
    "os.remove(\"checkpoints/pong-agent-test-policy.pkl\")\n",
    "print(\"Test passed!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Run this cell without modification\n",
    "\n",
    "pong_ac_config = dict(\n",
    "    # environment\n",
    "    env_name=\"PongDeterministic-v4\",\n",
    "\n",
    "    # model\n",
    "    hidden_units=100,\n",
    "\n",
    "    # training inner loop\n",
    "    learning_rate=0.0005,\n",
    "\n",
    "    # training outer loop\n",
    "    max_iteration=200,\n",
    "    max_episode_length=5000,  # In fact the game end at 10000 steps\n",
    "    train_batch_size=5000,\n",
    "\n",
    "    # evaluation\n",
    "    evaluate_interval=10,\n",
    "    evaluate_num_episodes=10\n",
    ")\n",
    "\n",
    "ac_trainer_pong, ac_result_pong = run(pong_wrapper(ActorCriticTrainer), pong_ac_config, 18.0)\n",
    "\n",
    "# Hint: Performance should greater than -20.0 within one hour.\n",
    "# We do not require you to solve the task, but you need to make your agent achieve > -20 reward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surfix = \"iter130\"\n",
    "\n",
    "ac_trainer_pong_restored = pong_wrapper(ActorCriticTrainer)(pong_ac_config)\n",
    "ac_trainer_pong_restored.restore(surfix)\n",
    "\n",
    "print(\"Average episode reward for your Actor-critic agent in PongDeterministic-v4: \",\n",
    "      ac_trainer_pong_restored.evaluate(1, render=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------\n",
    "\n",
    "## Conclusion and Discussion\n",
    "\n",
    "In this assignment, we implement several policy gradient algorithms.\n",
    "\n",
    "It's OK to leave the following cells empty. In the next markdown cell, you can write whatever you like. Like the suggestion on the course, the confusing problems in the assignments, and so on.\n",
    "\n",
    "If you want to do more investigation, feel free to open new cells via `Esc + B` after the next cells and write codes in it, so that you can reuse some result in this notebook. Remember to write sufficient comments and documents to let others know what you are doing.\n",
    "\n",
    "Following the submission instruction in the assignment to submit your assignment to our staff. Thank you!\n",
    "\n",
    "------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "torch",
   "language": "python",
   "name": "torch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
